[XL] Change in dart/sdk[main]: [vm/docs] Improve VM docs tooling.
56 views
Skip to first unread message
Slava Egorov (Gerrit)
Feb 15, 2023, 8:46:29 AM2/15/23
to Martin Kustermann, rev...@dartlang.org, vm-...@dartlang.org
Attention is currently required from: Martin Kustermann.
Slava Egorov would like Martin Kustermann to review this change.
[vm/docs] Improve VM docs tooling.
Main change is around removing our custom syntax which allows
to read markdown files directly on GitHub:
* instead of using custom link format `@{xref}` we start
using normal links. For example, [`dart::ThreadPool`][] is
understood as a ref to `dart::ThreadPool` class declaration.
`build.py` script injects an xref section at the end of
each markdown file.
* similarly we don't use custom syntax for admonitions, but
instead use blockquotes. `build.py` detects block quotes
which start with a marker like `**Note**` and renders
then in a custom way.
This CL also drops dependency on cquery and instead rewrites
indexing in pure Python via libclang.
Change-Id: I0b47ec93f632de89627a3c682d511c8b86c58430
---
D runtime/docs/images/aot-ic-dictionary.png
D runtime/docs/images/aot-ic-linear.png
D runtime/docs/images/aot-ic-monomorphic.png
D runtime/docs/images/aot-ic-singletarget.png
D runtime/docs/images/aot-ic-unlinked.png
D runtime/docs/images/aot.png
D runtime/docs/images/dart-to-kernel.png
D runtime/docs/images/flutter-cfe.png
D runtime/docs/images/images.graffle
D runtime/docs/images/inline-cache-1.png
D runtime/docs/images/isolates.png
D runtime/docs/images/kernel-loaded-1.png
D runtime/docs/images/kernel-loaded-2.png
D runtime/docs/images/kernel-service.png
D runtime/docs/images/optimizing-compilation.png
D runtime/docs/images/raw-function-lazy-compile.png
D runtime/docs/images/snapshot-appjit.png
D runtime/docs/images/snapshot-with-code.png
D runtime/docs/images/snapshot.png
D runtime/docs/images/unoptimized-compilation.png
M runtime/docs/index.md
D runtime/tools/wiki/CustomShellSessionPygmentsLexer/custom_shell_session/__init__.py
D runtime/tools/wiki/CustomShellSessionPygmentsLexer/custom_shell_session/lexer.py
D runtime/tools/wiki/CustomShellSessionPygmentsLexer/setup.py
M runtime/tools/wiki/README.md
A runtime/tools/wiki/build/admonitions.py
M runtime/tools/wiki/build/build.py
A runtime/tools/wiki/build/cpp_indexer.py
M runtime/tools/wiki/build/xrefs.py
M runtime/tools/wiki/styles/style.scss
D runtime/tools/wiki/templates/includes/favicon.html
M runtime/tools/wiki/templates/page.html
D runtime/tools/wiki/xref_extractor/.gitignore
D runtime/tools/wiki/xref_extractor/README.md
D runtime/tools/wiki/xref_extractor/analysis_options.yaml
D runtime/tools/wiki/xref_extractor/bin/main.dart
D runtime/tools/wiki/xref_extractor/lib/cquery_driver.dart
D runtime/tools/wiki/xref_extractor/lib/xref_extractor.dart
D runtime/tools/wiki/xref_extractor/pubspec.yaml
39 files changed, 1,523 insertions(+), 1,176 deletions(-)
diff --git a/runtime/docs/images/aot-ic-dictionary.png b/runtime/docs/images/aot-ic-dictionary.png
deleted file mode 100644
index 0010089..0000000
--- a/runtime/docs/images/aot-ic-dictionary.png
+++ /dev/null
Binary files differ
diff --git a/runtime/docs/images/aot-ic-linear.png b/runtime/docs/images/aot-ic-linear.png
deleted file mode 100644
index 4ab8b4e..0000000
--- a/runtime/docs/images/aot-ic-linear.png
+++ /dev/null
Binary files differ
diff --git a/runtime/docs/images/aot-ic-monomorphic.png b/runtime/docs/images/aot-ic-monomorphic.png
deleted file mode 100644
index a2e25da..0000000
--- a/runtime/docs/images/aot-ic-monomorphic.png
+++ /dev/null
Binary files differ
diff --git a/runtime/docs/images/aot-ic-singletarget.png b/runtime/docs/images/aot-ic-singletarget.png
deleted file mode 100644
index 4aed0c5..0000000
--- a/runtime/docs/images/aot-ic-singletarget.png
+++ /dev/null
Binary files differ
diff --git a/runtime/docs/images/aot-ic-unlinked.png b/runtime/docs/images/aot-ic-unlinked.png
deleted file mode 100644
index bb764c6..0000000
--- a/runtime/docs/images/aot-ic-unlinked.png
+++ /dev/null
Binary files differ
diff --git a/runtime/docs/images/aot.png b/runtime/docs/images/aot.png
deleted file mode 100644
index 1b777fd..0000000
--- a/runtime/docs/images/aot.png
+++ /dev/null
Binary files differ
diff --git a/runtime/docs/images/dart-to-kernel.png b/runtime/docs/images/dart-to-kernel.png
deleted file mode 100644
index 129626c..0000000
--- a/runtime/docs/images/dart-to-kernel.png
+++ /dev/null
Binary files differ
diff --git a/runtime/docs/images/flutter-cfe.png b/runtime/docs/images/flutter-cfe.png
deleted file mode 100644
index cf67b39..0000000
--- a/runtime/docs/images/flutter-cfe.png
+++ /dev/null
Binary files differ
diff --git a/runtime/docs/images/images.graffle b/runtime/docs/images/images.graffle
deleted file mode 100644
index fb77688..0000000
--- a/runtime/docs/images/images.graffle
+++ /dev/null
Binary files differ
diff --git a/runtime/docs/images/inline-cache-1.png b/runtime/docs/images/inline-cache-1.png
deleted file mode 100644
index 0edf3b6..0000000
--- a/runtime/docs/images/inline-cache-1.png
+++ /dev/null
Binary files differ
diff --git a/runtime/docs/images/isolates.png b/runtime/docs/images/isolates.png
deleted file mode 100644
index 13da2c6..0000000
--- a/runtime/docs/images/isolates.png
+++ /dev/null
Binary files differ
diff --git a/runtime/docs/images/kernel-loaded-1.png b/runtime/docs/images/kernel-loaded-1.png
deleted file mode 100644
index 7acce6e..0000000
--- a/runtime/docs/images/kernel-loaded-1.png
+++ /dev/null
Binary files differ
diff --git a/runtime/docs/images/kernel-loaded-2.png b/runtime/docs/images/kernel-loaded-2.png
deleted file mode 100644
index 2580701..0000000
--- a/runtime/docs/images/kernel-loaded-2.png
+++ /dev/null
Binary files differ
diff --git a/runtime/docs/images/kernel-service.png b/runtime/docs/images/kernel-service.png
deleted file mode 100644
index 7e19145..0000000
--- a/runtime/docs/images/kernel-service.png
+++ /dev/null
Binary files differ
diff --git a/runtime/docs/images/optimizing-compilation.png b/runtime/docs/images/optimizing-compilation.png
deleted file mode 100644
index 080df28..0000000
--- a/runtime/docs/images/optimizing-compilation.png
+++ /dev/null
Binary files differ
diff --git a/runtime/docs/images/raw-function-lazy-compile.png b/runtime/docs/images/raw-function-lazy-compile.png
deleted file mode 100644
index 39d69ac..0000000
--- a/runtime/docs/images/raw-function-lazy-compile.png
+++ /dev/null
Binary files differ
diff --git a/runtime/docs/images/snapshot-appjit.png b/runtime/docs/images/snapshot-appjit.png
deleted file mode 100644
index 0afd9ba..0000000
--- a/runtime/docs/images/snapshot-appjit.png
+++ /dev/null
Binary files differ
diff --git a/runtime/docs/images/snapshot-with-code.png b/runtime/docs/images/snapshot-with-code.png
deleted file mode 100644
index b6da0c5..0000000
--- a/runtime/docs/images/snapshot-with-code.png
+++ /dev/null
Binary files differ
diff --git a/runtime/docs/images/snapshot.png b/runtime/docs/images/snapshot.png
deleted file mode 100644
index 14d71a0..0000000
--- a/runtime/docs/images/snapshot.png
+++ /dev/null
Binary files differ
diff --git a/runtime/docs/images/unoptimized-compilation.png b/runtime/docs/images/unoptimized-compilation.png
deleted file mode 100644
index aabfa0b..0000000
--- a/runtime/docs/images/unoptimized-compilation.png
+++ /dev/null
Binary files differ
diff --git a/runtime/docs/index.md b/runtime/docs/index.md
index 21a9843..4f8089a 100644
--- a/runtime/docs/index.md
+++ b/runtime/docs/index.md
@@ -1,11 +1,12 @@
# Introduction to Dart VM
+> **Warning**
+>
+> This document is work in progress. Please contact [Vyacheslav Egorov](veg...@google.com) if you have any questions, suggestions, bug reports. **Last update: Oct 6 2022**
-!!! WARNING
- This document is work in progress and is currently being written. Please contact Vyacheslav Egorov ([by mail](veg...@google.com) or [@mraleph](http://twitter.com/mraleph)) if you have any questions, suggestions, bug reports. **Last update: January 29 2020**
+[TOC]
-!!! sourcecode "Purpose of this document"
- This document is intended as a reference for new members of the Dart VM team, potential external contributors or just anybody interested in VM internals. It starts with a high-level overview of the Dart VM and then proceeds to describe various components of the VM in more details.
+This document is intended as a reference for new members of the Dart VM team, potential external contributors or just anybody interested in VM internals. It starts with a high-level overview of the Dart VM and then proceeds to describe various components of the VM in more details.
Dart VM is a collection of components for executing Dart code natively. Notably it includes the following:
@@ -15,9 +16,9 @@
* Snapshots
* Core libraries native methods
* Development Experience components accessible via *service protocol*
- * Debugging
- * Profiling
- * Hot-reload
+ * Debugging
+ * Profiling
+ * Hot-reload
* Just-in-Time (JIT) and Ahead-of-Time (AOT) compilation pipelines
* Interpreter
* ARM simulators
@@ -35,9 +36,39 @@
However the main difference between these lies in when and how VM converts Dart source code to executable code. The runtime environment that facilitates the execution remains the same.
-
+```
+ pseudo isolate for
+ shared immutable objects
+ like null, true, false.
+ ┌────────────┐
+ │ VM Isolate │ heaps can reference
+ │ ╭────────╮ │ vm-isolate heap.
+ ┏━━━━━━━━━▶│ Heap │◀━━━━━━━━━━━━━━━━┓
+ ┃ │ ╰────────╯ │ ┃
+ ┃ └────────────┘ ┃
+ ┃ ┃
+ ┌─────────────────────────┃────────┐ ┌───────────────┃──────────────────┐
+ │ IsolateGroup ┃ │ │ IsolateGroup ┃ │
+ │ ┃ │ │ ┃ │
+ │ ╭───────────────────────┃──────╮ │ │ ╭─────────────┃────────────────╮ │
+ │ │ GC managed Heap ┅┅┅┅┅┅┅┅╳┅┅┅┅┅┅┅▶│ GC managed Heap │ │
+ │ ╰──────────────────────────────╯ │ no cross │ ╰──────────────────────────────╯ │
+ │ ┌─────────┐ ┌─────────┐ │ group │ ┌─────────┐ ┌─────────┐ │
+ │ │┌─────────┐ │┌─────────┐ │ references │ │┌─────────┐ │┌─────────┐ │
+ │ ││┌─────────┐ ││┌─────────┐ │ │ ││┌─────────┐ ││┌─────────┐ │
+ │ │││Isolate │ │││ │ │ │ │││Isolate │ │││ │ │
+ │ │││ │ │││ │ │ │ │││ │ │││ │ │
+ │ │││ globals │ │││ helper │ │ │ │││ globals │ │││ helper │ │
+ │ │││ │ │││ thread │ │ │ │││ │ │││ thread │ │
+ │ └││ mutator │ └││ │ │ │ └││ mutator │ └││ │ │
+ │ └│ thread │ └│ │ │ │ └│ thread │ └│ │ │
+ │ └─────────┘ └─────────┘ │ │ └─────────┘ └─────────┘ │
+ └──────────────────────────────────┘ └──────────────────────────────────┘
+```
-Any Dart code within the VM is running within some _isolate_, which can be best described as an isolated Dart universe with its own memory (*heap*) and _usually_ with its own thread of control (*mutator thread*). There can be many isolates executing Dart code concurrently, but they cannot share any state directly and can only communicate by message passing through *ports* (not to be confused with network ports!).
+Any Dart code within the VM is running within some _isolate_, which can be best described as an isolated Dart universe with its own global state and _usually_ with its own thread of control (*mutator thread*). Isolates are grouped together into _isolate groups_. Isolate within the group share the same garbage collector managed *heap*, used as a storage for objects allocated by an isolate. Heap sharing between isolates in the same group is an implementation detail which is not observable from the Dart code. Even isolates within the same group can not share any mutable state directly and can only communicate by message passing through *ports* (not to be confused with network ports!).
+
+Isolates within a group share the same Dart program. [`Isolate.spawn`](https://api.dart.dev/stable/dart-isolate/Isolate/spawn.html) spawns an isolate within the same group, while [`Isolate.spawnUri`](https://api.dart.dev/stable/2.18.2/dart-isolate/Isolate/spawnUri.html) starts a new group.
The relationship between OS threads and isolates is a bit blurry and highly dependent on how VM is embedded into an application. Only the following is guaranteed:
@@ -52,10 +83,11 @@
* GC sweeper threads;
* concurrent GC marker threads.
-Internally VM uses a thread pool (@{dart::ThreadPool}) to manage OS threads and the code is structured around @{dart::ThreadPool::Task} concept rather than around a concept of an OS thread. For example, instead of spawning a dedicated thread to perform background sweeping after a GC VM posts a @{dart::ConcurrentSweeperTask} to the global VM thread pool and thread pool implementation either selects an idling thread or spawns a new thread if no threads are available. Similarly the default implementation of an event loop for isolate message processing does not actually spawn a dedicated event loop thread, instead it posts a @{dart::MessageHandlerTask} to the thread pool whenever a new message arrives.
+Internally VM uses a thread pool [`dart::ThreadPool`][] to manage OS threads and the code is structured around [`dart::ThreadPool::Task`][] concept rather than around a concept of an OS thread. For example, instead of spawning a dedicated thread to perform background sweeping after a GC VM posts a [`dart::ConcurrentSweeperTask`][] to the global VM thread pool and thread pool implementation either selects an idling thread or spawns a new thread if no threads are available. Similarly the default implementation of an event loop for isolate message processing does not actually spawn a dedicated event loop thread, instead it posts a [`dart::MessageHandlerTask`][] to the thread pool whenever a new message arrives.
-!!! sourcecode "Source to read"
- Class @{dart::Isolate} represents an isolate, class @{dart::Heap} - isolate's heap. Class @{dart::Thread} describes the state associated with a thread attached to an isolate. Note that the name `Thread` is somewhat confusing because all OS threads attached to the same isolate as a mutator would reuse the same `Thread` instance. See @{Dart_RunLoop} and @{dart::MessageHandler} for the default implementation of an isolate's message handling.
+> **Source to read**
+>
+> Class [`dart::Isolate`][] represents an isolate, [`dart::IsolateGroup`][] an isolate group and class [`dart::Heap`][] - isolate group's heap. Class [`dart::Thread`][] describes the state associated with a thread attached to an isolate. Note that the name `Thread` is somewhat confusing because all OS threads attached to the same isolate as a mutator will reuse the same `Thread` instance. See [`Dart_RunLoop`][] and [`dart::MessageHandler`][] for the default implementation of an isolate's message handling.
### Running from source via JIT.
@@ -71,74 +103,178 @@
Hello, World!
```
-Since Dart 2 VM no longer has the ability to directly execute Dart from raw source, instead VM expects to be given _Kernel binaries_ (also called _dill files_) which contain serialized [Kernel ASTs][what-is-kernel]. The task of translating Dart source into Kernel AST is handled by the [common front-end (CFE)][pkg-front_end] written in Dart and shared between different Dart tools (e.g. VM, dart2js, Dart Dev Compiler).
+Since Dart 2 VM no longer has the ability to directly execute Dart from raw source, instead VM expects to be given _Kernel binaries_ (also called _dill files_) which contain serialized [Kernel ASTs][`pkg/kernel/README.md`]. The task of translating Dart source into Kernel AST is handled by the [common front-end (CFE)][`pkg/front_end`] written in Dart and shared between different Dart tools (e.g. VM, dart2js, Dart Dev Compiler).
-
+```
+ ╭─────────────╮ ╭────────────╮
+ │╭─────────────╮ ╔═════╗ │╭────────────╮ ╔════╗
+ ││╭─────────────╮┣━━━▶ ║ CFE ║ ┣━━━▶ ││╭────────────╮ ┣━━━▶ ║ VM ║
+ ┆││ Dart Source │ ╚═════╝ │││ Kernel AST │ ╚════╝
+ ┆┆│ │ ╰││ (binary) │
+ ┆┆ ┆ ╰│ │
+ ┆ ┆ ╰────────────╯
+```
-To preserve convenience of executing Dart directly from source standalone `dart` executable hosts a helper isolate called *kernel service*, which handles compilation of Dart source into Kernel. VM then would run resulting Kernel binary.
+To preserve convenience of executing Dart directly from source standalone `dart` executable hosts a helper isolate called *kernel service*, which handles compilation of Dart source into Kernel. VM then will run resulting Kernel binary.
-
+```
+ ┌───────────────────────────────────────────────────┐
+ │ dart (cli) │
+ │ ┌─────────┐ ┌─────────┐ │
+ ╭─────────────╮ │ │ kernel │ ╭────────────╮ │ main │ │
+ │╭─────────────╮ │ │ service │ │╭────────────╮ │ isolate │ │
+ ││╭─────────────╮┣━━━▶│ isolate │┣━━━▶││╭────────────╮┣━━━▶│ │ │
+ ┆││ Dart Source │ │ │ │ │││ Kernel AST │ │ │ │
+ ┆┆│ │ │ │╔══════╗ │ ╰││ (binary) │ │ │ │
+ ┆┆ ┆ │ │║ CFE ║ │ ╰│ │ │ │ │
+ ┆ ┆ │ │╚══════╝ │ ╰────────────╯ │ │ │
+ │ │ │══════════════════════════│ │ │
+ │ └─────────┘ VM └─────────┘ │
+ │ ╚═════════════════════════════════╝ │
+ └───────────────────────────────────────────────────┘
+```
+
However this setup is not the only way to arrange CFE and VM to run Dart code. For example, Flutter completely separates _compilation to Kernel_ and _execution from Kernel_ by putting them onto different devices: compilation happens on the developer machine (_host_) and execution is handled on the target mobile _device_, which receives Kernel binaries send to it by `flutter` tool.
-
+```
+ HOST ┆ DEVICE
+ ┆
+ ┌──────────────────────┐ ┆ ┌────────────────┐
+ ╭─────────────╮ │frontend_server (CFE) │ ┆ │ Flutter Engine │
+ │╭─────────────╮ ┌──────────────────────┐ │ ┆ │ ╔════════════╗ │
+ ││╭─────────────╮┣━━━▶│flutter run --debug │ │ ┆ │ ║ VM ║ │
+ ┆││ Dart Source │ │ │─┘ ┆ │ ╚════════════╝ │
+ ┆┆│ │ │ │ ┆ └────────────────┘
+ ┆┆ ┆ └──────────────────────┘ ┆ ▲
+ ┆ ┆ ┳ ┆ ┃
+ ┃ ┆ ┃
+ ┃ ╭────────────╮ ┃
+ ┃ │╭────────────╮ ┃
+ ┃ ││╭────────────╮ ┃
+ ┗━━━━━━━━▶│││ Kernel AST │┣━━━┛
+ ╰││ (binary) │
+ ╰│ │
+ ╰────────────╯
+```
Note that `flutter` tool does not handle parsing of Dart itself - instead it spawns another persistent process `frontend_server`, which is essentially a thin wrapper around CFE and some Flutter specific Kernel-to-Kernel transformations. `frontend_server` compiles Dart source into Kernel files, which `flutter` tool then sends to the device. Persistence of the `frontend_server` process comes into play when developer requests _hot reload_: in this case `frontend_server` can reuse CFE state from the previous compilation and recompile just libraries which actually changed.
Once Kernel binary is loaded into the VM it is parsed to create objects representing various program entities. However this is done lazily: at first only basic information about libraries and classes is loaded. Each entity originating from a Kernel binary keeps a pointer back to the binary, so that later more information can be loaded as needed.
-<aside>We use `Raw...` prefix whenever we talk about specific objects allocated internally by the VM. This follows VM own naming convention: layout of internal VM objects is defined using C++ classes with names starting with `Raw` in the header file @{runtime/vm/raw_object.h}. For example, @{dart::RawClass} is a VM object describing Dart class, @{dart::RawField} is a VM object describing a Dart field within a Dart class and so on. We will return to this in a section covering runtime system and object model.</aside>
+> **Note**
+>
+> Definitions of internal VM objects, like those representing classes and functions, are split into two parts: class `Xyz` in the header [`runtime/vm/object.h`][] defines C++ methods, while class `UntaggedXyz` in the header [`runtime/vm/raw_object.h`][] defines memory layout. For example, [`dart::Class`][] and [`dart::UntaggedClass`][] specify a VM object describing Dart class, [`dart::Field`][] and [`dart::UntaggedField`][] specify a VM object describing a Dart field within a Dart class and so on. We will return to this in a section covering runtime system and object model. We omit `Untagged...` prefix from illustrations to make them more compact.
-
+```
+ KERNEL AST BINARY ┆ ISOLATE GROUP HEAP
+ ┆
+ ╭─────────────────╮ ┆ ┌───────┐
+ │ │ ┆ ┏━┥ Class │
+ ├─────────────────┤ ┆ ┃ └───────┘╲ heap objects
+ AST node │(Class │◀━━┛ representing
+ representing │ (Field) │ ┆ ┌───────┐╱ a class
+ a class │ (Procedure │ ┆ ┏━┥ Class │
+ │ (FunctionNode))│ ┆ ┃ └───────┘
+ │ (Procedure │ ┆ ┃
+ │ (FunctionNode))│ ┆ ┃╲
+ ├─────────────────┤ ┆ ┃ ╲ heap objects representing
+ │(Class │◀━━┛ program entities keep
+ │ (Field) │ ┆ pointers back into kernel
+ │ (Field) │ ┆ binary blob and are
+ │ (Procedure │ ┆ deserialized lazily
+ │ (FunctionNode))│ ┆
+ ├─────────────────┤ ┆
+ ┆
+```
Information about the class is fully deserialized only when runtime later needs it (e.g. to lookup a class member, to allocate an instance, etc). At this stage class members are read from the Kernel binary. However full function bodies are not deserialized at this stage, only their signatures.
-
+```
+ KERNEL AST BINARY ┆ ISOLATE GROUP HEAP
+ ┆
+ ╭─────────────────╮ ┆ ┌───────┐
+ │ │ ┆ ┏━┥ Class ┝━━━━━━━┓
+ ├─────────────────┤ ┆ ┃ └───────┘ ┃
+ │(Class │◀━━┛ ┌──────────┐ ┃
+ │ (Field) │◀━━━━━━┥ Field │◀━┫
+ │ (Procedure │◀━━━━┓ └──────────┘ ┃
+ │ (FunctionNode))│ ┆ ┃ ┌──────────┐ ┃
+ │ (Procedure │◀━━━┓┗━┥ Function │◀━┫
+ │ (FunctionNode))│ ┆ ┃ └──────────┘ ┃
+ ├─────────────────┤ ┆ ┃ ┌──────────┐ ┃
+ │(Class │ ┆ ┗━━┥ Function │◀━┛
+ │ (Field) │ ┆ └──────────┘
+ │ (Field) │ ┆
+ │ (Procedure │ ┆
+ │ (FunctionNode))│ ┆
+ ├─────────────────┤ ┆
+ ┆
+```
At this point enough information is loaded from Kernel binary for runtime to successfully resolve and invoke methods. For example, it could resolve and invoke `main` function from a library.
-!!! sourcecode "Source to read"
- @{package:kernel/ast.dart} defines classes describing the Kernel AST. @{package:front_end} handles parsing Dart source and building Kernel AST from it. @{dart::kernel::KernelLoader::LoadEntireProgram} is an entry point for deserialization of Kernel AST into corresponding VM objects. @{pkg/vm/bin/kernel_service.dart} implements the Kernel Service isolate, @{runtime/vm/kernel_isolate.cc} glues Dart implementation to the rest of the VM. @{package:vm} hosts most of the Kernel based VM specific functionality, e.g various Kernel-to-Kernel transformations.
+> **Source to read**
+>
+> [`package:kernel/ast.dart`][] defines classes describing the Kernel AST. [`package:front_end`][] handles parsing Dart source and building Kernel AST from it. [`dart::kernel::KernelLoader::LoadEntireProgram`][] is an entry point for deserialization of Kernel AST into corresponding VM objects. [`pkg/vm/bin/kernel_service.dart`][] implements the Kernel Service isolate, [`runtime/vm/kernel_isolate.cc`][] glues Dart implementation to the rest of the VM. [`package:vm`][] hosts most of the Kernel based VM specific functionality, e.g various Kernel-to-Kernel transformations.
-!!! tryit "Trying it"
- If you are interested in Kernel format and its VM specific usage, then you can use @{pkg/vm/bin/gen_kernel.dart} to produce a Kernel binary file from Dart source. Resulting binary can then be dumped using @{pkg/vm/bin/dump_kernel.dart}.
-
- ```custom-shell-session
- # Take hello.dart and compile it to hello.dill Kernel binary using CFE.
- $ dart pkg/vm/bin/gen_kernel.dart \
- --platform out/ReleaseX64/vm_platform_strong.dill \
- -o hello.dill \
- hello.dart
-
- # Dump textual representation of Kernel AST.
- $ dart pkg/vm/bin/dump_kernel.dart hello.dill hello.kernel.txt
- ```
-
- When you try using `gen_kernel.dart` you will notice that it requires something called *platform*, a Kernel binary containing AST for all core libraries (`dart:core`, `dart:async`, etc). If you have Dart SDK build configured then you can just use platform file from the `out` directory, e.g. `out/ReleaseX64/vm_platform_strong.dill`. Alternatively you can use
- @{pkg/front_end/tool/_fasta/compile_platform.dart} to generate the platform:
-
- ```custom-shell-session
- # Produce outline and platform files using the given libraries list.
- $ dart pkg/front_end/tool/_fasta/compile_platform.dart \
- dart:core \
- sdk/lib/libraries.json \
- vm_outline.dill vm_platform.dill vm_outline.dill
- ```
+> **Trying it**
+>
+> If you are interested in Kernel format and its VM specific usage, then you can use [`pkg/vm/bin/gen_kernel.dart`][] to produce a Kernel binary file from Dart source. Resulting binary can then be dumped using [`pkg/vm/bin/dump_kernel.dart`][].
+>
+> * Compile `hello.dart` to `hello.dill` Kernel binary using CFE
+> ```console
+> $ dart pkg/vm/bin/gen_kernel.dart \
+> --platform out/ReleaseX64/vm_platform_strong.dill \
+> -o hello.dill \
+> hello.dart
+> ```
+> * Dump textual representation of Kernel AST.
+> ```console
+> $ dart pkg/vm/bin/dump_kernel.dart hello.dill hello.kernel.txt
+> ```
+>
+> When you try using `gen_kernel.dart` you will notice that it requires something called *platform*, a Kernel binary containing AST for all core libraries (`dart:core`, `dart:async`, etc). If you have Dart SDK build configured then you can just use platform file from the `out` directory, e.g. `out/ReleaseX64/vm_platform_strong.dill`. Alternatively you can use [`pkg/front_end/tool/_fasta/compile_platform.dart`][] to generate the platform:
+>
+> ```console
+> $ dart pkg/front_end/tool/_fasta/compile_platform.dart \
+> dart:core \
+> sdk/lib/libraries.json \
+> vm_outline.dill vm_platform.dill vm_outline.dill
+> ```
Initially all functions have a placeholder instead of an actually executable code for their bodies: they point to `LazyCompileStub`, which simply asks runtime system to generate executable code for the current function and then tail-calls this newly generated code.
-
+```
+┌──────────┐
+│ Function │
+│ │ LazyCompileStub
+│ code_ ━━━━━━▶ ┌─────────────────────────────┐
+│ │ │ code = CompileFunction(...) │
+└──────────┘ │ return code(...); │
+ └─────────────────────────────┘
+```
When the function is compiled for the first time this is done by *unoptimizing compiler*.
-
+```
+ Kernel AST Unoptimized IL Machine Code
+╭──────────────╮ ╭──────────────────╮ ╭─────────────────────╮
+│ FunctionNode │ │ LoadLocal('a') │ │ push [rbp + ...] │
+│ │ │ LoadLocal('b') │ │ push [rbp + ...] │
+│ (a, b) => │ ┣━━▶ │ InstanceCall('+')│ ┣━━▶ │ call InlineCacheStub│
+│ a + b; │ │ Return │ │ retq │
+╰──────────────╯ ╰──────────────────╯ ╰─────────────────────╯
+```
Unoptimizing compiler produces machine code in two passes:
1. Serialized AST for the function's body is walked to generate a *control flow graph* (**CFG**) for the function body. CFG consists of basic blocks filled with *intermediate language* (**IL**) instructions. IL instructions used at this stage resemble instructions of a stack based virtual machine: they take operands from the stack, perform operations and then push results to the same stack.
-<aside>In reality not all functions have actual Dart / Kernel AST bodies, e.g. *natives* defined in C++ or artificial tear-off functions generated by Dart VM - in these cases IL is just created from the thin air, instead of being generated from Kernel AST.</aside>
+> **Note**
+>
+> In reality not all functions have actual Dart / Kernel AST bodies, e.g. *natives* defined in C++ or artificial tear-off functions generated by Dart VM - in these cases IL is just created from the thin air, instead of being generated from Kernel AST.
2. resulting CFG is directly compiled to machine code using one-to-many lowering of IL instructions: each IL instruction expands to multiple machine language instructions.
@@ -148,14 +284,38 @@
The core idea behind inline caching is to cache results of method resolution in a call site specific cache. Inline caching mechanism used by the VM consists of:
-<aside>Original implementations of inline caching were actually patching the native code of the function - hence the name _**inline** caching_. The idea of inline caching dates far back to Smalltalk-80, see [Efficient implementation of the Smalltalk-80 system](https://dl.acm.org/citation.cfm?id=800542).</aside>
+> **Note**
+>
+> Original implementations of inline caching were actually patching the native code of the function - hence the name _**inline** caching_. The idea of inline caching dates far back to Smalltalk-80, see [Efficient implementation of the Smalltalk-80 system](https://dl.acm.org/citation.cfm?id=800542).
-* a call site specific cache (@{dart::RawICData} object) that maps receiver's class to a method, that should be invoked if receiver is of a matching class. The cache also stores some auxiliary information, e.g. invocation frequency counters, which track how often the given class was seen at this call site;
-* a shared lookup stub, which implements method invocation fast path. This stub searches through the given cache to see if it contains an entry that matches receiver's class. If the entry is found then stub would increment the frequency counter and tail call cached method. Otherwise stub would invoke a runtime system helper which implements method resolution logic. If method resolution succeeds then cache would be updated and subsequent invocations would not need to enter runtime system.
+* a call site specific cache ([`dart::UntaggedICData`][] object) that maps receiver's class to a method, that should be invoked if receiver is of a matching class. The cache also stores some auxiliary information, e.g. invocation frequency counters, which track how often the given class was seen at this call site;
+* a shared lookup stub, which implements method invocation fast path. This stub searches through the given cache to see if it contains an entry that matches receiver's class. If the entry is found then stub increments the frequency counter and tail call cached method. Otherwise stub invokes a runtime system helper which implements method resolution logic. If method resolution succeeds then cache is updated and subsequent invocations will not need to enter runtime system.
The picture below illustrates the structure and the state of an inline cache associated with `animal.toFace()` call site, which was executed twice with an instance of `Dog` and once with an instance of a `Cat`.
-
+```
+class Dog {
+ get face => '🐶';
+}
+
+class Cat {
+ get face => '🐱';
+} ICData
+ ┌─────────────────────────────────────┐
+sameFace(animal, face) ┌─────────▶│// class, method, frequency │
+ object.face == face; │ │[Dog, Dog.get:face, 2, │
+ ┬ │ │ Cat, Cat.get:face, 1] │
+ └──────────────┤ └─────────────────────────────────────┘
+sameFace(Dog(), ...); │ InlineCacheStub
+sameFace(Dog(), ...); │ ┌─────────────────────────────────────┐
+sameFace(Cat(), ...); └─────────▶│ idx = cache.indexOf(classOf(this)); │
+ │ if (idx != -1) { │
+ │ cache[idx + 2]++; // frequency++ │
+ │ return cache[idx + 1](...); │
+ │ } │
+ │ return InlineCacheMiss(...); │
+ └─────────────────────────────────────┘
+```
Unoptimizing compiler by itself is enough to execute any possible Dart code. However the code it produces is rather slow, which is why VM also implements *adaptive optimizing* compilation pipeline. The idea behind adaptive optimization is to use execution profile of a running program to drive optimization decisions.
@@ -171,39 +331,73 @@
Once compilation is complete background compiler requests mutator thread to enter a *safepoint* and attaches optimized code to the function.
-<aside>Broadly speaking a thread in a managed environment (virtual machine) is considered to be at a *safepoint* when the state associated with it (e.g. stack frames, heap, etc) is consistent and can be accessed or modified without interruption from the thread itself. Usually this implies that the thread is either paused or is executing some code outside of the managed environment e.g. running unmanaged native code. See [GC](/gc.html) page for more information.</aside>
+> **Note**
+>
+> Broadly speaking a thread in a managed environment (virtual machine) is considered to be at a *safepoint* when the state associated with it (e.g. stack frames, heap, etc) is consistent and can be accessed or modified without interruption from the thread itself. Usually this implies that the thread is either paused or is executing some code outside of the managed environment e.g. running unmanaged native code. See [GC](/gc.html) page for more information.
The next time this function is called - it will use optimized code. Some functions contain very long running loops and for those it makes sense to switch execution from unoptimized to optimized code while the function is still running. This process is called *on stack replacement* (**OSR**) owing its name to the fact that a stack frame for one version of the function is transparently replaced with a stack frame for another version of the same function.
-
+```
+ in hot code ICs
+ Kernel AST Unoptimized IL have collected type
+╭──────────────╮ ╭───────────────────────╮ ╱ feedback
+│ FunctionNode │ │ LoadLocal('a') │ ICData
+│ │ │ LoadLocal('b') ┌────▶┌─────────────────────┐
+│ (a, b) => │ ┣━━▶ │ InstanceCall:1('+', ┴)│ │[(Smi, Smi.+, 10000)]│
+│ a + b; │ │ Return ╱ │ └─────────────────────┘
+╰──────────────╯ ╰────────────╱──────────╯
+ deopt id ┳
+ ┃
+ SSA IL ▼
+ ╭────────────────────────────────╮
+ │ v1<-Parameter('a') │
+ │ v2<-Parameter('b') │
+ │ v3<-InstanceCall:1('+', v1, v2)│
+ │ Return(v3) │
+ ╰────────────────────────────────╯
+ ┳
+ ┃
+ Machine Code ▼ Optimized SSA IL
+ ╭─────────────────────╮ ╭──────────────────────────────╮
+ │ movq rax, [rbp+...] │ │ v1<-Parameter('a') │
+ │ testq rax, 1 │ ◀━━┫│ v2<-Parameter('b') │
+ │ jnz ->deopt@1 │ │ CheckSmi:1(v1) │
+ │ movq rbx, [rbp+...] │ │ CheckSmi:1(v2) │
+ │ testq rbx, 1 │ │ v3<-BinarySmiOp:1(+, v1, v2) │
+ │ jnz ->deopt@1 │ │ Return(v3) │
+ │ addq rax, rbx │ ╰──────────────────────────────╯
+ │ jo ->deopt@1 │
+ │ retq │
+ ╰─────────────────────╯
+```
-!!! sourcecode "Source to read"
- Compiler sources are in the @{runtime/vm/compiler} directory.
- Compilation pipeline entry point is @{dart::CompileParsedFunctionHelper::Compile}. IL is defined in @{runtime/vm/compiler/backend/il.h}. Kernel-to-IL translation starts in @{dart::kernel::StreamingFlowGraphBuilder::BuildGraph}, and this function also handles construction of IL for various artificial functions. @{dart::compiler::StubCodeCompiler::GenerateNArgsCheckInlineCacheStub} generates machine code for inline-cache stub, while @{InlineCacheMissHandler} handles IC misses. @{runtime/vm/compiler/compiler_pass.cc} defines optimizing compiler passes and their order. @{dart::JitCallSpecializer} does most of the type-feedback based specializations.
+> **Source to read**
+>
+> Compiler sources are in the [`runtime/vm/compiler`][] directory.
+> Compilation pipeline entry point is [`dart::CompileParsedFunctionHelper::Compile`][]. IL is defined in [`runtime/vm/compiler/backend/il.h`][]. Kernel-to-IL translation starts in [`dart::kernel::StreamingFlowGraphBuilder::BuildGraph`][], and this function also handles construction of IL for various artificial functions. [`dart::compiler::StubCodeCompiler::GenerateNArgsCheckInlineCacheStub`][] generates machine code for inline-cache stub, while [`dart::InlineCacheMissHandler`][] handles IC misses. [`runtime/vm/compiler/compiler_pass.cc`][] defines optimizing compiler passes and their order. [`dart::JitCallSpecializer`][] does most of the type-feedback based specializations.
-!!! tryit "Trying it"
- VM also has flags which can be used to control JIT and to make it dump IL and generated machine code for the functions that are being compiled by the JIT.
-
- | Flag | Description |
- | ---- | ---- |
- | `--print-flow-graph[-optimized]` | Print IL for all (or only optimized) compilations |
- | `--disassemble[-optimized]` | Disassemble all (or only optimized) compiled functions |
- | `--print-flow-graph-filter=xyz,abc,...` | Restrict output triggered by previous flags only to the functions which contain one of the comma separated substrings in their names |
- | `--compiler-passes=...` | Fine control over compiler passes: force IL to be printed before/after a certain pass. Disable passes by name. Pass `help` for more information |
- | `--no-background-compilation` | Disable background compilation, and compile all hot functions on the main thread. Useful for experimentation, otherwise short running programs might finish before background compiler compiles hot function |
-
- For example
-
- ```myshell
- # Run test.dart and dump optimized IL and machine code for
- # function(s) that contain(s) "myFunction" in its name.
- # Disable background compilation for determinism.
- $ dart --print-flow-graph-optimized \
- --disassemble-optimized \
- --print-flow-graph-filter=myFunction \
- --no-background-compilation \
- test.dart
- ```
+> **Trying it**
+>
+> VM also has flags which can be used to control JIT and to make it dump IL and generated machine code for the functions that are being compiled by the JIT.
+>
+> | Flag | Description |
+> | ---- | ---- |
+> | `--print-flow-graph[-optimized]` | Print IL for all (or only optimized) compilations |
+> | `--disassemble[-optimized]` | Disassemble all (or only optimized) compiled functions |
+> | `--print-flow-graph-filter=xyz,abc,...` | Restrict output triggered by previous flags only to the functions which contain one of the comma separated substrings in their names |
+> | `--compiler-passes=...` | Fine control over compiler passes: force IL to be printed before/after a certain pass. Disable passes by name. Pass `help` for more information |
+> | `--no-background-compilation` | Disable background compilation, and compile all hot functions on the main thread. Useful for experimentation, otherwise short running programs might finish before background compiler compiles hot function |
+> | `--deterministic` | Disable various sources of non-determinism in the VM (concurrent GC and compiler). |
+>
+> For example the following command will run `test.dart` and dump optimized IL and machine code for functions that contain `myFunction` in their names:
+>
+> ```console
+> $ dart --print-flow-graph-optimized \
+> --disassemble-optimized \
+> --print-flow-graph-filter=myFunction \
+> --no-background-compilation \
+> test.dart
+> ```
It is important to highlight that the code generated by optimizing compiler is specialized under speculative assumptions based on the execution profile of the application. For example, a dynamic call site that only observed instances of a single class `C` as a receiver will be converted into a direct call preceded by a check verifying that receiver has an expected class `C`. However these assumptions might be violated later during execution of the program:
@@ -233,7 +427,10 @@
This process of recovery is known as _deoptimization_: whenever optimized version hits a case which it can't handle, it simply transfers execution into the matching point of unoptimized function and continues execution there. Unoptimized version of a function does not make any assumptions and can handle all possible inputs.
-<aside>Entering unoptimized function at the right spot is absolutely crucial because code has side-effects (e.g. in the function above deoptimization happens after we already executed the first `print`). Matching instructions that deoptimize to positions in the unoptimized code in VM is done using *deopt ids*</aside>
+
+> **Note**
+>
+> Entering unoptimized function at the right spot is absolutely crucial because code has side-effects (e.g. in the function above deoptimization happens after we already executed the first `print`). Matching instructions that deoptimize to positions in the unoptimized code in VM is done using *deopt ids*
VM usually discards optimized version of the function after deoptimization and
then reoptimizes it again later - using updated type feedback.
@@ -241,32 +438,68 @@
There are two ways VM guards speculative assumptions made by the compiler:
* Inline checks (e.g. `CheckSmi`, `CheckClass` IL instructions) that verify if assumption holds at *use* site where compiler made this assumption. For example, when turning dynamic calls into direct calls compiler adds these checks right before a direct call. Deoptimization that happens on such checks is called *eager deoptimization*, because it occurs eagerly as the check is reached.
-* Global guards which instruct runtime to discard optimized code when it changes something that optimized code relies on. For example, optimizing compiler might observe that some class `C` is never extended and use this information during type propagation pass. However subsequent dynamic code loading or class finalization can introduce a subclass of `C` - which invalidates the assumption. At this point runtime needs to find and discard all optimized code that was compiled under the assumption that `C` has no subclasses. It is possible that runtime would find some of the now invalid optimized code on the execution stack - in which case affected frames would be marked for deoptimization and will deoptimize when execution returns to them. This sort of deoptimization is called *lazy deoptimization*: because it is delayed until control returns back to the optimized code.
+* Global guards which instruct runtime to discard optimized code when it changes something that optimized code relies on. For example, optimizing compiler might observe that some class `C` is never extended and use this information during type propagation pass. However subsequent dynamic code loading or class finalization can introduce a subclass of `C` - which invalidates the assumption. At this point runtime needs to find and discard all optimized code that was compiled under the assumption that `C` has no subclasses. It is possible that runtime finds some of the now invalid optimized code on the execution stack - in which case affected frames are marked for deoptimization and will deoptimize when execution returns to them. This sort of deoptimization is called *lazy deoptimization*: because it is delayed until control returns back to the optimized code.
-!!! sourcecode "Source to read"
- Deoptimizer machinery resides in @{runtime/vm/deopt_instructions.cc}. It is essentially a mini-interpreter for *deoptimization instructions* which describe how to reconstruct needed state of the unoptimized code from the state of optimized code. Deoptimization instructions are generated by @{dart::CompilerDeoptInfo::CreateDeoptInfo} for every potential deoptimization location in optimized code during compilation.
+> **Source to read**
+>
+> Deoptimizer machinery resides in [`runtime/vm/deopt_instructions.cc`][]. It is essentially a mini-interpreter for *deoptimization instructions* which describe how to reconstruct needed state of the unoptimized code from the state of optimized code. Deoptimization instructions are generated by [`dart::CompilerDeoptInfo::CreateDeoptInfo`][] for every potential deoptimization location in optimized code during compilation.
-!!! tryit "Trying it"
- Flag `--trace-deoptimization` makes VM print information about the cause and location of every deoptimization that occurs. `--trace-deoptimization-verbose` makes VM print a line for every deoptimization instruction it executes during deoptimization.
+> **Trying it**
+>
+> Flag `--trace-deoptimization` makes VM print information about the cause and location of every deoptimization that occurs. `--trace-deoptimization-verbose` makes VM print a line for every deoptimization instruction it executes during deoptimization.
### Running from Snapshots
VM has the ability to serialize isolate's heap or more precisely object graph residing in the heap into a binary *snapshot*. Snapshot then can be used to recreate the same state when starting VM isolates.
-
+```
+ ┌──────────────┐
+ SNAPSHOT ┌──────────────┐│
+┌──────────────┐ ╭─────────╮ ┌──────────────┐││
+│ HEAP │ │ 0101110 │ │ HEAP │││
+│ ██ │ │ 1011010 │ │ ██ │││
+│ ╱ ╲ │ │ 1010110 │ │ ╱ ╲ │││
+│ ██╲ ██ │ │ 1101010 │ │ ██╲ ██ │││
+│ ╲ ╱ ╲ │┣━━━━━━━━━▶│ 0010101 │┣━━━━━━━━━▶│ ╲ ╱ ╲ │││
+│ ╳ ██ │ serialize │ 0101011 │deserialize│ ╳ ██ │││
+│ ╱ ╲ ╱ │ │ 1111010 │ │ ╱ ╲ ╱ │││
+│ ██ ██ │ │ 0010110 │ │ ██ ██ ││┘
+│ │ │ 0001011 │ │ │┘
+└──────────────┘ ╰─────────╯ └──────────────┘
+
+```
Snapshot's format is low level and optimized for fast startup - it is essentially a list of objects to create and instructions on how to connect them together. That was the original idea behind snapshots: instead of parsing Dart source and gradually creating internal VM data structures, VM can just spin an isolate up with all necessary data structures quickly unpacked from the snapshot.
-<aside>The idea of a snapshots has roots in Smalltalk [images](https://en.wikipedia.org/wiki/Smalltalk#Image-based_persistence) which were in turn inspired by [Alan Kay's M.Sc thesis](https://www.mprove.de/visionreality/media/kay68.html). Dart VM is using clustered serialization format which is similar to techniques described in [Parcels: a Fast and Feature-Rich Binary Deployment Technology](http://scg.unibe.ch/archive/papers/Mira05aParcels.pdf) and [Clustered serialization with Fuel](https://rmod.inria.fr/archives/workshops/Dia11a-IWST11-Fuel.pdf) papers.</aside>
+> **Note**
+>
+> The idea of a snapshots has roots in Smalltalk [images](https://en.wikipedia.org/wiki/Smalltalk#Image-based_persistence) which were in turn inspired by [Alan Kay's M.Sc thesis](https://www.mprove.de/visionreality/media/kay68.html). Dart VM is using clustered serialization format which is similar to techniques described in [Parcels: a Fast and Feature-Rich Binary Deployment Technology](http://scg.unibe.ch/archive/papers/Mira05aParcels.pdf) and [Clustered serialization with Fuel](https://rmod.inria.fr/archives/workshops/Dia11a-IWST11-Fuel.pdf) papers.
Initially snapshots did not include machine code, however this capability was later added when AOT compiler was developed. Motivation for developing AOT compiler and snapshots-with-code was to allow VM to be used on the platforms where JITing is impossible due to platform level restrictions.
Snapshots-with-code work almost in the same way as normal snapshots with a minor difference: they include a code section which unlike the rest of the snapshot does not require deserialization. This code section laid in way that allows it to directly become part of the heap after it was mapped into memory.
-
+```
+ ┌──────────────┐
+ SNAPSHOT ┌──────────────┐│
+┌──────────────┐ ╭────╮╭────╮ ┌──────────────┐││
+│ HEAP │ │ 01 ││ │ │ HEAP │││
+│ ██ │ │ 10 ││ ░░◀──┐ │ ██ │││
+│ ╱ ╲ │ │ 10 ││ │ │ │ ╱ ╲ │││
+│ ██╲ ██ │ │ 11 ││ │ └────────────██╲ ██ │││
+│ ╱ ╲ ╱ ╲ │┣━━━━━━━━━▶│ 00 ││ │┣━━━━━━━━━▶│ ╲ ╱ ╲ │││
+│ ░░ ╳ ██ │ serialize │ 01 ││ │deserialize│ ╳ ██ │││
+│ ╱ ╲ ╱ │ │ 11 ││ │ │ ╱ ╲ ╱ │││
+│ ██ ██╲ │ │ 00 ││ │ │ ██ ██ ││┘
+│ ░░ │ │ 00 ││ ░░◀──────────────────────┘ │┘
+└──────────╱───┘ ╰────╯╰────╯ └──────────────┘
+ code data code
+```
-!!! sourcecode "Source to read"
- @{runtime/vm/app_snapshot.cc} handles serialization and deserialization of snapshots. A family of API functions `Dart_CreateXyzSnapshot[AsAssembly]` are responsible for writing out snapshots of the heap (e.g. @{Dart_CreateAppJITSnapshotAsBlobs} and @{Dart_CreateAppAOTSnapshotAsAssembly}). On the other hand @{Dart_CreateIsolateGroup} optionally takes snapshot data to start an isolate from.
+
+> **Source to read**
+>
+> [`runtime/vm/app_snapshot.cc`][] handles serialization and deserialization of snapshots. A family of API functions `Dart_CreateXyzSnapshot[AsAssembly]` are responsible for writing out snapshots of the heap (e.g. [`Dart_CreateAppJITSnapshotAsBlobs`][] and [`Dart_CreateAppAOTSnapshotAsAssembly`][]). On the other hand [`Dart_CreateIsolateGroup`][] optionally takes snapshot data to start an isolate from.
### Running from AppJIT snapshots
@@ -274,34 +507,50 @@
AppJIT snapshots allow to address this problem: an application can be run on the VM using some mock training data and then all generated code and VM internal data structures are serialized into an AppJIT snapshot. This snapshot can then be distributed instead of distributing application in the source (or Kernel binary) form. VM starting from this snapshot can still JIT - if it turns out that execution profile on the real data does not match execution profile observed during training.
-
+```
+ ┌──────────────┐
+ SNAPSHOT ┌──────────────┐│
+┌──────────────┐ ╭────╮╭────╮ ┌──────────────┐││
+│ HEAP │ │ 01 ││ │ │ HEAP │││
+│ ██ │ │ 10 ││ ░░◀──┐ │ ██ │││
+│ ╱ ╲ │ │ 10 ││ │ │ │ ╱ ╲ │││
+│ ██╲ ██ │ │ 11 ││ │ └────────────██╲ ██ │││
+│ ╱ ╲ ╱ ╲ │┣━━━━━━━━━▶│ 00 ││ │┣━━━━━━━━━▶│ ╲ ╱ ╲ │││
+│ ░░ ╳ ██ │ serialize │ 01 ││ │deserialize│ ╳ ██ │││
+│ ╱ ╲ ╱ │ │ 11 ││ │ │ ╱ ╲ ╱│ │││
+│ ██ ██╲ │ │ 00 ││ │ │ ██ ██ │ ││┘
+│ ░░ │ │ 00 ││ ░░◀──────────────────────┘ ░░ │┘
+└──────────╱───┘ ╰────╯╰────╯ └──────────╱───┘
+ code data code isolate can JIT more
+```
-!!! tryit "Trying it"
- `dart` binary will generate AppJIT snapshot after running the application if you pass `--snapshot-kind=app-jit --snapshot=path-to-snapshot` to it. Here is an example of generating and using an AppJIT snapshot for `dart2js`.
-
- ```custom-shell-session
- # Run from source in JIT mode.
- $ dart pkg/compiler/lib/src/dart2js.dart -o hello.js hello.dart
- Compiled 7,359,592 characters Dart to 10,620 characters JavaScript in 2.07 seconds
- Dart file (hello.dart) compiled to JavaScript: hello.js
-
- # Training run to generate app-jit snapshot
- $ dart --snapshot-kind=app-jit --snapshot=dart2js.snapshot \
- pkg/compiler/lib/src/dart2js.dart -o hello.js hello.dart
- Compiled 7,359,592 characters Dart to 10,620 characters JavaScript in 2.05 seconds
- Dart file (hello.dart) compiled to JavaScript: hello.js
-
- # Run from app-jit snapshot.
- $ dart dart2js.snapshot -o hello.js hello.dart
- Compiled 7,359,592 characters Dart to 10,620 characters JavaScript in 0.73 seconds
- Dart file (hello.dart) compiled to JavaScript: hello.js
- ```
+> **Trying it**
+>
+> `dart` binary will generate AppJIT snapshot after running the application if you pass `--snapshot-kind=app-jit --snapshot=path-to-snapshot` to it. Here is an example of generating and using an AppJIT snapshot for `dart2js`.
+>
+> * Run from source in JIT mode
+> ```console
+> $ dart pkg/compiler/lib/src/dart2js.dart -o hello.js hello.dart
+> Dart file (hello.dart) compiled to JavaScript: hello.js
+> ```
+> * Create an app-jit snapshot trained by compiling `dart2js` with itself, then
+> run from this snapshot.
+> ```console
+> $ dart --snapshot-kind=app-jit --snapshot=dart2js.snapshot \
+> pkg/compiler/lib/src/dart2js.dart -o hello.js hello.dart
+> Dart file (hello.dart) compiled to JavaScript: hello.js
+>
+> $ dart dart2js.snapshot -o hello.js hello.dart
+> Dart file (hello.dart) compiled to JavaScript: hello.js
+> ```
### Running from AppAOT snapshots
AOT snapshots were originally introduced for platforms which make JIT compilation impossible, but they can also be used in situations where fast startup and consistent performance is worth potential peak performance penalty.
-<aside>There is usually a lot of confusion around how performance characteristics of JIT and AOT compare. JIT has access to precise local type information and execution profile of the running application, however it has to pay for it with warmup. AOT can infer and prove various properties globally (for which it has to pay with compile time), but has no information of how the program will actually be executing - on the other hand AOT compiled code reaches its peak performance almost immediately with virtual no warmup. Currently Dart VM JIT has best peak performance, while Dart VM AOT has best startup time.</aside>
+> **Note**
+>
+> There is usually a lot of confusion around how performance characteristics of JIT and AOT compare. JIT has access to precise local type information and execution profile of the running application, however it has to pay for it with warmup. AOT can infer and prove various properties globally (for which it has to pay with compile time), but has no information of how the program will actually be executing - on the other hand AOT compiled code reaches its peak performance almost immediately with virtual no warmup. Currently Dart VM JIT has best peak performance, while Dart VM AOT has best startup time.
Inability to JIT implies that:
@@ -316,78 +565,396 @@
Resulting snapshot can then be run using *precompiled runtime*, a special variant of the Dart VM which excludes components like JIT and dynamic code loading facilities.
-
+```
+ ╭─────────────╮ ╭────────────╮
+ │╭─────────────╮ ╔═════╗ │ Kernel AST │
+ ││╭─────────────╮┣━━━▶ ║ CFE ║ ┣━━━▶ │ │
+ ┆││ Dart Source │ ╚═════╝ │ whole │
+ ┆┆│ │ │ program │
+ ┆┆ ┆ ╰────────────╯
+ ┆ ┆ ┳
+ ┃
+ ▼
+ ╔═════╗ type-flow analysis
+ ║ TFA ║ propagates types globally
+ VM contains an ╚═════╝ through the whole program
+ AOT compilation ┳
+ pipeline which ┃
+ reuses parts of ▼
+ JIT pipeline ╭────────────╮
+ ╭────────────╮ ╲ │ Kernel AST │
+ │AOT Snapshot│ ╔════╗ │ │
+ │ │◀━━━┫ ║ VM ║ ◀━━━┫ │ inferred │
+ │ │ ╚════╝ │ treeshaken │
+ ╰────────────╯ ╰────────────╯
+```
-!!! sourcecode "Source to read"
- @{package:vm/transformations/type_flow/transformer.dart} is an entry point to the type flow analysis and transformation based on TFA results. @{dart::Precompiler::DoCompileAll} is an entry point to the AOT compilation loop in the VM.
+> **Source to read**
+>
+> [`package:vm/transformations/type_flow/transformer.dart`][] is an entry point to the type flow analysis and transformation based on TFA results. [`dart::Precompiler::DoCompileAll`][] is an entry point to the AOT compilation loop in the VM.
-!!! tryit "Trying it"
- AOT compilation pipeline is currently packaged into Dart SDK as [`dart2native` script](https://dart.dev/tools/dart2native).
-
- ```custom-shell-session
- $ dart2native hello.dart -o hello
- $ ./hello
- Hello, World!
- ```
-
- Note that it is impossible to pass options like `--print-flow-graph-optimized` and `--disassemble-optimized` to the `dart2native` script so if you would like to inspect generated AOT code you will need to build compiler from source.
-
- ```custom-shell-session
- # Need to build normal dart executable and runtime for running AOT code.
- $ tool/build.py -m release -a x64 runtime dart_precompiled_runtime
-
- # Now compile an application using AOT compiler
- $ pkg/vm/tool/precompiler2 hello.dart hello.aot
-
- # Execute AOT snapshot using runtime for AOT code
- $ out/ReleaseX64/dart_precompiled_runtime hello.aot
- Hello, World!
- ```
-
-#### Switchable Calls
-
-Even with global and local analyses AOT compiled code might still contain call sites which could not be *devirtualized* (meaning they could not be statically resolved). To compensate for this AOT compiled code and runtime use an extension of inline caching technique utilized in JIT. This extended version is called *switchable calls*.
-
-JIT section already described that each inline cache associated with a call site consists of two pieces: a cache object (represented by an instance of @{dart::RawICData}) and a chunk of native code to invoke (e.g. a @{dart::compiler::StubCodeCompiler::GenerateNArgsCheckInlineCacheStub|InlineCacheStub}). In JIT mode runtime would only update the cache itself. However in AOT runtime can choose to replace both the cache and the native code to invoke depending on the state of the inline cache.
-
-
-
-Initially all dynamic calls start in the *unlinked* state. When such call-site is reached for the first time @{dart::compiler::StubCodeCompiler::GenerateUnlinkedCallStub|UnlinkedCallStub} is invoked, which simply calls into runtime helper @{DRT_UnlinkedCall} to link this call site.
-
-If possible @{DRT_UnlinkedCall} tries to transition the call site into a _monomorphic_ state. In this state call site turns into a direct call, which enters method through a special entry point which verifies that receiver has expected class.
-
-
-
-In the example above we assume that when `obj.method()` was executed for the first time `obj` was an instance of `C` and `obj.method` resolved to `C.method`.
-
-Next time we execute the same call-site it would invoke `C.method` directly, bypassing any sort of method lookup process. However it would enter `C.method` through a special entry point, which would verify that `obj` is still an instance of `C`. If that is not the case @{DRT_MonomorphicMiss} would be invoked and will try to select the next call site state.
-
-`C.method` might still be a valid target for an invocation, e.g `obj` is an instance of the class `D` which extends `C` but does not override `C.method`. In this case we check if call site could transition into a *single target* state, implemented by @{dart::compiler::StubCodeCompiler::GenerateSingleTargetCallStub|SingleTargetCallStub} (see also @{dart::RawSingleTargetCache}).
-
-
-
-This stub is based on the fact that for AOT compilation most classes are assigned integer ids using depth-first traversal of the inheritance hierarchy. If `C` is a base class with subclasses `D0, ..., Dn` and none of those override `C.method` then `C.:cid <= classId(obj) <= max(D0.:cid, ..., Dn.:cid)` implies that `obj.method` resolves to `C.method`. In this circumstances instead of comparing to a single class (*monomorphic* state), we can use class id range check (*single target* state) which would work for all subclasses of `C`.
-
-Otherwise call site would be switched to use linear search inline cache, similar to the one used in JIT mode (see @{dart::compiler::StubCodeCompiler::GenerateICCallThroughCodeStub|ICCallThroughCodeStub}, @{dart::RawICData} and @{DRT_MegamorphicCacheMissHandler}).
-
-
-
-Finally if the number of checks in the linear array grows past threshold the call site is switched to use a dictionary like structure (see @{dart::compiler::StubCodeCompiler::GenerateMegamorphicCallStub|MegamorphicCallStub}, @{dart::RawMegamorphicCache} and @{DRT_MegamorphicCacheMissHandler}).
-
-
-
-[what-is-kernel]: https://github.com/dart-lang/sdk/blob/main/pkg/kernel/README.md
-[pkg-front_end]: https://github.com/dart-lang/sdk/tree/master/pkg/front_end
+> **Trying it**
+>
+> AOT compilation pipeline is currently packaged into Dart SDK as [`dart compile exe` command](https://dart.dev/tools/dart2native).
+>
+> ```console
+> $ dart compile exe -o hello hello.dart
+> $ ./hello
+> Hello, World!
+> ```
+>
+> It is possible to pass options like `--print-flow-graph-optimized` and `--disassemble-optimized` to the `dart compile exe` via `--extra-gen-snapshot-options` flag. You also need to pass `--verbose` to see the output, otherwise it is silently swallowed by the tool.
+>
+> ```console
+> $ dart compile exe --verbose \
+> --extra-gen-snapshot-options=--print-flow-graph-optimized \
+> --extra-gen-snapshot-options=--print-flow-graph-filter=main \
+> --extra-gen-snapshot-options=--disassemble \
+> hello.dart
+> ```
+>
## Runtime System
-
-!!! WARNING
- This section will be written next.
-
### Object Model
-## TODO
+### Representation of Types
-1. Document the difference between CoreJIT and AppJIT snapshots.
-2. Document that switchable calls are used in the unoptimized code as well.
+See [Representation of Types](types.md).
+
+### GC
+
+See [GC](gc.md).
+
+## Compiler
+
+### Method Calls
+
+There is currently large difference between how AOT and JIT optimize method invocation sequences.
+
+JIT keeps to its Dart 1 roots and largely ignores statically typed nature of Dart 2. In unoptimized code method calls by default go through an inline cache which collects type feedback. Optimizing compiler then speculatively specializes indirect method calls into direct calls guarded by _class checks_. This process is called _speculative devirtualization_. Those call sites which can't be devirtualized are divided into two categories. Those call sites which have not been executed yet are compiled to use inline caching and collect type feedback for subsequent reoptimizations. Those call sites which are highly polymorphic (megamorphic) are compiled to use metamorphic dispatch.
+
+On the other hand, AOT heavily leans onto the statically typed nature of Dart 2. The compiler uses results of global type flow analysis (TFA) to devirtualize as many call sites as it can. This devirtualization is not speculative: compiler only devirtualizes the call site if it can prove that it always invokes a specific method. If compiler can not devirtualize a call site, then it chooses a dispatch mechanism based on whether the receiver's static type is `dynamic` or not. Calls on `dynamic` receiver use switchable calls. All other calls go through a _global dispatch table_.
+
+#### Global Dispatch Table (GDT)
+
+> **Note**
+>
+> The approach adopted by Dart VM and described in this section is largely based on insights from [Minimizing row displacement dispatch tables](https://dl.acm.org/doi/abs/10.1145/217839.217851) by Karel Driesen and Urs Holzle.
+
+Imagine for a moment that each class defined in the program added its methods to a global dictionary. For example, given the following class hierachy
+
+```dart
+class A {
+ void foo() { }
+ void bar() { }
+}
+
+class B extends A {
+ void foo() { }
+ void baz() { }
+}
+```
+
+This dictionary will contain the following:
+
+```dart
+globalDispatchTable = {
+ // Calling [foo] on an instance of [A] hits [A.foo].
+ (A, #foo): A.foo,
+ // Calling [bar] on an instance of [A] hits [A.bar].
+ (A, #bar): A.bar,
+ // Calling [foo] on an instance of [B] hits [B.foo].
+ (B, #foo): B.foo,
+ // Calling [bar] on an instance of [B] hits [A.bar].
+ (B, #bar): A.bar,
+ // Calling [baz] on an instance of [B] hits [B.baz].
+ (B, #baz): B.baz
+};
+```
+
+Compiler could then use such a dictionary to dispatch invocations: a method call `o.m(...)` will be compiled into `globalDispatchTable[(classOf(o), #m)](o, ...)`.
+
+A naive approach to representing `globalDispatchTable` (or `gdt` for short) is to number all classes and all method selectors in the program sequentially and then use a two-dimensional array: `gdt[(classOf(o), #m)]` becomes `gdt[o.cid][#m.id]`. At this point we can choose to flatten this two-dimensional array either using selector-major order (`gdt[numClasses * #m.id + o.cid]`) or class-major order (`gdt[numSelectors * o.cid + #m.id]`) .
+
+Let us take a look at selector-major order. In this representation we say that `numClasses * #m.id` gives us _selector offset_: an offset into the GDT at which a row of entries (one per class) corresponding to this selector is stored. Consider the following class hierarchy:
+
+```dart
+class A {
+ void foo() { }
+}
+
+class B extends A {
+ void foo() { }
+}
+
+class C {
+ void bar() { }
+}
+
+class D extends C {
+ void bar() { }
+}
+```
+
+Classes `A`, `B`, `C` and `D` will be numbered 0, 1, 2 and 3 respectively, while selectors `foo` and `bar` will be numbered `0` and `1`. This will lead to the following array:
+
+```
+offset 0 4
+ │A B C D │
+ ┌─────┬─────┬─────┬─────┐
+foo row │A.foo│B.foo│ NSM │ NSM │
+ └─────┴─────┴─────┴─────┘
+ │A B C D
+ ┌─────┬─────┬─────┬─────┐
+bar row │ NSM │ NSM │C.bar│D.bar│
+ └─────┴─────┴─────┴─────┘
+ ┌─────┬─────┬─────┬─────┬─────┬─────┬─────┬─────┐
+GDT │A.foo│B.foo│ NSM │ NSM │ NSM │ NSM │C.bar│D.bar│
+ └─────┴─────┴─────┴─────┴─────┴─────┴─────┴─────┘
+```
+
+It's evident that such representation is rather memory inefficient: dispatch table ends up with a lot of NSM (`noSuchMethod`) entries.
+
+Fortunately, Dart 2 static type system provides us with a way to compress this table. In Dart 2 static type of the receiver constrains the list of selectors allowed by the compiler. This guarantees that any non-`dynamic` invocation calls an actual method rather than `Object.noSuchMethod`. Consequently, if we only use dispatch table for non-`dynamic` call sites then we don't need to fill holes in the table with NSM entries.
+
+This leads to the following idea: instead of numbering selectors sequentially and using `numClasses * sid` as a selector offset, we could instead select selector offsets which causes selector rows to interleave and reuse available holes.
+
+Let us look back to the previous example with 4 classes. Instead of numbering `foo` with `0` and `bar` with `1` and using `0` and `4` as a selector offsets respectively, we could simply assign both selectors an offset of `0` leading to the following compact table
+
+```
+offset 0
+ │A B C D
+ ┌─────┬─────┬─────┬─────┐
+foo row │A.foo│B.foo│░░░░░│░░░░░│
+ └─────┴─────┴─────┴─────╲
+ A B C D hole
+ ┌─────┬─────┬─────┬─────┐
+bar row │░░░░░│░░░░░│C.bar│D.bar│
+ └─────┴─────┴─────┴─────┘
+ ┌─────┬─────┬─────┬─────┐
+GDT │A.foo│B.foo│C.bar│D.bar│
+ └─────┴─────┴─────┴─────┘
+```
+
+This works because it is impossible to invoke `bar` on `A` or `B` and it's impossible to invoke `foo` on `C` or `D` - meaning, for example, that `A.foo` entry will never be hit with an instance of `C` as receiver.
+
+> **Note**
+>
+> Strictly speaking this technique does not actually require static typing and was originally applied to Smalltalk. This requires a small trick: each method should check in prologue whether its selector matches the selector of the invocation which led to this method being invoked. If selectors don't match this means we arrived to this method erroneously through a reused NSM entry. Static typing allows us to avoid this check.
+
+Calls through GDT compile to the following machine code in Dart VM (X64 example):
+
+```nasm
+movzx cid, word ptr [obj + 15] ; load receiver's class id
+call [GDT + cid * 8 + (selectorOffset - 16) * 8]
+```
+
+Here `GDT` is a reserved register containing a biased pointer to the GDT (`&GDT[16]` on X64) and `selectorOffset` is an offset of the selector we are invoking. The call looks similar across architectures, though concrete value of the bias (specified by [`dart::DispatchTable::kOriginElement`][]) depends on the target architecture. We bias `GDT` pointer to have a more compact call sequence encoding for smaller selectors, e.g. on X64
+an indirect `call` has an encoding which allows for an 1 byte signed immediate offset. This means that imeddiate offsets in the range `-128` to `127` are represented as a single byte. With an unbiased GDT pointer we would only be able to utilize half of this range because `selectorOffset` is an unsigned value. With biased GDT we can use the full range:`selectorOffset` `15` still requires just one byte encoding.
+
+> **Source to read**
+>
+> Computation of the global dispatch table is spread through different
+> parts of the toolchain.
+>
+> * [`TableSelectorAssigner`][`package:vm/transformations/type_flow/table_selector_assigner.dart`] is responsible for assigning selector ids to methods in the program.
+> * [`dart::DispatchTableCallInstr`][] is an IL instruction representing a call through
+> GDT.
+> * [`dart::AotCallSpecializer::ReplaceInstanceCallsWithDispatchTableCalls`][] is a compiler pass
+> which replaces non-devirtualized method calls with GDT calls.
+> * [`dart::FlowGraphCompiler::EmitDispatchTableCall`][] emits architecture specific
+> call sequence for calls through GDT.
+> * [`dart::compiler::DispatchTableGenerator`][] is responsible for assigning
+> selector offsets and computing final layout of the table.
+
+#### Switchable Calls
+
+Switchable call is an extension of an inline caching originally developed for Dart 1 AOT - where they were used to compile all method calls. Current AOT only uses them when compiling calls with a `dynamic` receiver. They are also used in JIT to speedup calls from unoptimised code
+
+JIT section already described that each inline cache associated with a call site consists of two pieces: a cache object (represented by an instance of [`dart::UntaggedICData`][]) and a chunk of native code to invoke (e.g. an [inline cache stub][`dart::compiler::StubCodeCompiler::GenerateNArgsCheckInlineCacheStub`]). Original implementation in JIT was only updating the cache itself, however this was later extended by allowing runtime system to update both the cache and the stub target depending on the types observed by the call site.
+
+```
+ UnlinkedCall
+ cache ┌────────────────────────────────────┐
+ ┌─────────▶│targetName: "method" │
+ │ └────────────────────────────────────┘
+ ┴
+object.method()
+ ┬ SwitchableCallMissStub
+ │ ┌────────────────────────────────────┐
+ └─────────▶│ return DRT_SwitchableCallMiss(...);│
+ target └────────────────────────────────────┘
+```
+
+Initially all `dynamic` calls in AOT start in the *unlinked* state. When such call-site is reached for the first time [SwitchableCallMissStub][`dart::compiler::StubCodeCompiler::GenerateSwitchableCallMissStub`] is invoked, which simply calls into runtime helper [`dart::DRT_SwitchableCallMiss`][] to link this call site.
+
+If possible [`dart::DRT_SwitchableCallMiss`][] tries to transition the call site into a _monomorphic_ state. In this state call site turns into a direct call, which enters method through a special entry point which verifies that receiver has expected class.
+
+```
+
+ cache ┌────────────────────────────────────┐
+ ┌─────────▶│id of class C │
+ │ └────────────────────────────────────┘
+ ┴
+object.method()
+ ┬ C.method
+ │ ┌────────────────────────────────────┐
+ └─────────▶│ monomorphic_entry: │
+ target │ // Check if receiver's cid │
+ │ // matches the cached one. │
+ │ if (this.cid != cache) │
+ │ return SwitchableCallMissStub()│
+ normal calls │ // fall through to normal entry │
+ enter here ────▶│ normal_entry: │
+ │ // Body of C.method │
+ └────────────────────────────────────┘
+```
+
+In the example above we assume that `obj` was an instance of `C` and that `obj.method` resolved to `C.method` when `obj.method()` was executed for the first time.
+
+Next time we execute the same call-site it will invoke `C.method` directly bypassing method lookup process. However it will enter `C.method` through a special entry point, which will verify that `obj` is still an instance of `C`. If that is not the case [`dart::DRT_SwitchableCallMiss`][] will be invoked and will update call site state to reflect the miss.
+
+`C.method` might still be a valid target for an invocation, e.g `obj` is an instance of the class `D` which extends `C` but does not override `C.method`. In this case we check if call site could transition into a *single target* state, implemented by [SingleTargetCallStub][`dart::compiler::StubCodeCompiler::GenerateSingleTargetCallStub`] (see also [`dart::UntaggedSingleTargetCache`][]).
+
+```
+ SingleTargetCache
+ cache ┌────────────────────────────────────┐
+ ┌─────────▶│fromCid, toCid, target │
+ │ └────────────────────────────────────┘
+ ┴
+object.method() SingleTargetCallStub
+ ┬ ┌────────────────────────────────────┐
+ └─────────▶│ if (cache.fromCid <= this.cid && │
+ target │ this.cid <= cache.toCid ) │
+ │ return cache.target(...); │
+ │ // Not found │
+ │ return SwitchableCallMissStub(...);│
+ └────────────────────────────────────┘
+```
+
+
+This stub benefits from depth-first class id assignment done by AOT compiler and during AppJIT snapshot training. In this mode most classes are assigned integer ids using depth-first traversal of the inheritance hierarchy. If `C` is a base class with subclasses `D0, ..., Dn` and none of those override `C.method` then `C.:cid <= classId(obj) <= max(D0.:cid, ..., Dn.:cid)` implies that `obj.method` resolves to `C.method`. In such cases instead of comparing for equality (which checks a for a specific class), we can instead compare if class id falls into a specific range and that will cover all subclasses of `C`. That's exactly what `SingleTargetCallStub` does.
+
+If single target case is not applicable call site is switched to use linear search inline cache. Which coincidentally is also the initial state for call-sites in JIT mode (see [`ICCallThroughCode`][`dart::compiler::StubCodeCompiler::GenerateICCallThroughCodeStub`] stub, [`dart::UntaggedICData`][] and [`dart::DRT_SwitchableCallMiss`][]).
+
+```
+ ICData
+ cache ┌────────────────────────────────────┐
+ ┌─────────▶│{cid0: target0, cid1: target1, ... }│
+ │ └────────────────────────────────────┘
+ ┴
+object.method() ICCallThroughCodeStub
+ ┬ ┌────────────────────────────────────┐
+ └─────────▶│ for (i = 0; i < cache.length; i++) │
+ target │ if (cache[i] == this.cid) │
+ │ return cache[i + 1](...); │
+ │ // Not found │
+ │ return SwitchableCallMissStub(...);│
+ └────────────────────────────────────┘
+```
+
+
+Finally if the number of checks in the linear array grows past threshold the call site is switched to use a dictionary like structure (see [MegamorphicCallStub][`dart::compiler::StubCodeCompiler::GenerateMegamorphicCallStub`], [`dart::UntaggedMegamorphicCache`][] and [`dart::DRT_SwitchableCallMiss`][]).
+
+```
+ MegamorphicCache
+ cache ┌────────────────────────────────────┐
+ ┌─────────▶│{cid0: target0, cid1: target1, ... }│
+ │ └────────────────────────────────────┘
+ ┴
+object.method() MegamorphicCallStub
+ ┬ ┌────────────────────────────────────┐
+ └─────────▶│ var target = cache[this.cid]; │
+ target │ if (target != null) │
+ │ return target(...); │
+ │ // Not found │
+ │ return SwitchableCallMissStub(...);│
+ └────────────────────────────────────┘
+```
+
+### `try`/`catch` in IL
+
+See [Exceptions Implementation](compiler/exceptions.md).
+
+### `async`, `async*` and `sync*` methods
+
+See [Suspendable Functions](async.md).
+
+### `as` checks
+
+See [Type Testing Stubs](compiler/type_testing_stubs.md).
+
+## Miscellaneous
+
+### Pragmas
+
+See [VM-Specific Pragma Annotations](pragmas.md).
+
+### DWARF (non-symbolic) stack traces
+
+See [DWARF stack traces mode](dwarf_stack_traces.md).
+
+## Glossary
+
+See [Glossary](glossary.md)
+
+
+<!-- AUTOGENERATED XREF SECTION -->
+[`dart::ThreadPool`]: https://github.com/dart-lang/sdk/blob/2ed6ea29003476e2a28fb5f4683a656427eb41ff/runtime/vm/thread_pool.h#L20
+[`dart::ThreadPool::Task`]: https://github.com/dart-lang/sdk/blob/2ed6ea29003476e2a28fb5f4683a656427eb41ff/runtime/vm/thread_pool.h#L23
+[`dart::ConcurrentSweeperTask`]: https://github.com/dart-lang/sdk/blob/2ed6ea29003476e2a28fb5f4683a656427eb41ff/runtime/vm/heap/sweeper.cc#L109
+[`dart::MessageHandlerTask`]: https://github.com/dart-lang/sdk/blob/2ed6ea29003476e2a28fb5f4683a656427eb41ff/runtime/vm/message_handler.cc#L23
+[`pkg/kernel/README.md`]: https://github.com/dart-lang/sdk/blob/2ed6ea29003476e2a28fb5f4683a656427eb41ff/pkg/kernel/README.md
+[`pkg/front_end`]: https://github.com/dart-lang/sdk/blob/2ed6ea29003476e2a28fb5f4683a656427eb41ff/pkg/front_end
+[`runtime/vm/object.h`]: https://github.com/dart-lang/sdk/blob/2ed6ea29003476e2a28fb5f4683a656427eb41ff/runtime/vm/object.h
+[`runtime/vm/raw_object.h`]: https://github.com/dart-lang/sdk/blob/2ed6ea29003476e2a28fb5f4683a656427eb41ff/runtime/vm/raw_object.h
+[`dart::Class`]: https://github.com/dart-lang/sdk/blob/2ed6ea29003476e2a28fb5f4683a656427eb41ff/runtime/vm/object.h#L1027
+[`dart::UntaggedClass`]: https://github.com/dart-lang/sdk/blob/2ed6ea29003476e2a28fb5f4683a656427eb41ff/runtime/vm/raw_object.h#L958
+[`dart::Field`]: https://github.com/dart-lang/sdk/blob/2ed6ea29003476e2a28fb5f4683a656427eb41ff/runtime/vm/object.h#L4099
+[`dart::UntaggedField`]: https://github.com/dart-lang/sdk/blob/2ed6ea29003476e2a28fb5f4683a656427eb41ff/runtime/vm/raw_object.h#L1438
+[`dart::DispatchTable::kOriginElement`]: https://github.com/dart-lang/sdk/blob/2ed6ea29003476e2a28fb5f4683a656427eb41ff/runtime/vm/dispatch_table.h#L31
+[`dart::UntaggedICData`]: https://github.com/dart-lang/sdk/blob/2ed6ea29003476e2a28fb5f4683a656427eb41ff/runtime/vm/raw_object.h#L2453
+[`dart::compiler::StubCodeCompiler::GenerateNArgsCheckInlineCacheStub`]: https://github.com/dart-lang/sdk/blob/2ed6ea29003476e2a28fb5f4683a656427eb41ff/runtime/vm/compiler/stub_code_compiler_arm64.cc#L2471
+[`dart::compiler::StubCodeCompiler::GenerateSwitchableCallMissStub`]: https://github.com/dart-lang/sdk/blob/2ed6ea29003476e2a28fb5f4683a656427eb41ff/runtime/vm/compiler/stub_code_compiler_arm64.cc#L3613
+[`dart::DRT_SwitchableCallMiss`]: https://github.com/dart-lang/sdk/blob/2ed6ea29003476e2a28fb5f4683a656427eb41ff/runtime/vm/runtime_entry.cc#L2551
+[`dart::compiler::StubCodeCompiler::GenerateSingleTargetCallStub`]: https://github.com/dart-lang/sdk/blob/2ed6ea29003476e2a28fb5f4683a656427eb41ff/runtime/vm/compiler/stub_code_compiler_arm64.cc#L3640
+[`dart::UntaggedSingleTargetCache`]: https://github.com/dart-lang/sdk/blob/2ed6ea29003476e2a28fb5f4683a656427eb41ff/runtime/vm/raw_object.h#L2414
+[`dart::compiler::StubCodeCompiler::GenerateICCallThroughCodeStub`]: https://github.com/dart-lang/sdk/blob/2ed6ea29003476e2a28fb5f4683a656427eb41ff/runtime/vm/compiler/stub_code_compiler_arm64.cc#L3537
+[`dart::compiler::StubCodeCompiler::GenerateMegamorphicCallStub`]: https://github.com/dart-lang/sdk/blob/2ed6ea29003476e2a28fb5f4683a656427eb41ff/runtime/vm/compiler/stub_code_compiler_arm64.cc#L3456
+[`dart::UntaggedMegamorphicCache`]: https://github.com/dart-lang/sdk/blob/2ed6ea29003476e2a28fb5f4683a656427eb41ff/runtime/vm/raw_object.h#L2481
+[`package:vm/transformations/type_flow/table_selector_assigner.dart`]: https://github.com/dart-lang/sdk/blob/2ed6ea29003476e2a28fb5f4683a656427eb41ff/pkg/vm/lib/transformations/type_flow/table_selector_assigner.dart
+[`dart::DispatchTableCallInstr`]: https://github.com/dart-lang/sdk/blob/2ed6ea29003476e2a28fb5f4683a656427eb41ff/runtime/vm/compiler/backend/il.h#L4758
+[`dart::AotCallSpecializer::ReplaceInstanceCallsWithDispatchTableCalls`]: https://github.com/dart-lang/sdk/blob/2ed6ea29003476e2a28fb5f4683a656427eb41ff/runtime/vm/compiler/aot/aot_call_specializer.cc#L1128
+[`dart::FlowGraphCompiler::EmitDispatchTableCall`]: https://github.com/dart-lang/sdk/blob/2ed6ea29003476e2a28fb5f4683a656427eb41ff/runtime/vm/compiler/backend/flow_graph_compiler_arm64.cc#L665
+[`dart::compiler::DispatchTableGenerator`]: https://github.com/dart-lang/sdk/blob/2ed6ea29003476e2a28fb5f4683a656427eb41ff/runtime/vm/compiler/aot/dispatch_table_generator.h#L89
+[`package:vm/transformations/type_flow/transformer.dart`]: https://github.com/dart-lang/sdk/blob/2ed6ea29003476e2a28fb5f4683a656427eb41ff/pkg/vm/lib/transformations/type_flow/transformer.dart
+[`dart::Precompiler::DoCompileAll`]: https://github.com/dart-lang/sdk/blob/2ed6ea29003476e2a28fb5f4683a656427eb41ff/runtime/vm/compiler/aot/precompiler.cc#L452
+[`runtime/vm/app_snapshot.cc`]: https://github.com/dart-lang/sdk/blob/2ed6ea29003476e2a28fb5f4683a656427eb41ff/runtime/vm/app_snapshot.cc
+[`Dart_CreateAppJITSnapshotAsBlobs`]: https://github.com/dart-lang/sdk/blob/2ed6ea29003476e2a28fb5f4683a656427eb41ff/runtime/vm/dart_api_impl.cc#L6954
+[`Dart_CreateAppAOTSnapshotAsAssembly`]: https://github.com/dart-lang/sdk/blob/2ed6ea29003476e2a28fb5f4683a656427eb41ff/runtime/vm/dart_api_impl.cc#L6662
+[`Dart_CreateIsolateGroup`]: https://github.com/dart-lang/sdk/blob/2ed6ea29003476e2a28fb5f4683a656427eb41ff/runtime/vm/dart_api_impl.cc#L1375
+[`runtime/vm/deopt_instructions.cc`]: https://github.com/dart-lang/sdk/blob/2ed6ea29003476e2a28fb5f4683a656427eb41ff/runtime/vm/deopt_instructions.cc
+[`dart::CompilerDeoptInfo::CreateDeoptInfo`]: https://github.com/dart-lang/sdk/blob/2ed6ea29003476e2a28fb5f4683a656427eb41ff/runtime/vm/compiler/backend/flow_graph_compiler_arm64.cc#L87
+[`runtime/vm/compiler`]: https://github.com/dart-lang/sdk/blob/2ed6ea29003476e2a28fb5f4683a656427eb41ff/runtime/vm/compiler
+[`dart::CompileParsedFunctionHelper::Compile`]: https://github.com/dart-lang/sdk/blob/2ed6ea29003476e2a28fb5f4683a656427eb41ff/runtime/vm/compiler/jit/compiler.cc#L473
+[`runtime/vm/compiler/backend/il.h`]: https://github.com/dart-lang/sdk/blob/2ed6ea29003476e2a28fb5f4683a656427eb41ff/runtime/vm/compiler/backend/il.h
+[`dart::kernel::StreamingFlowGraphBuilder::BuildGraph`]: https://github.com/dart-lang/sdk/blob/2ed6ea29003476e2a28fb5f4683a656427eb41ff/runtime/vm/compiler/frontend/kernel_binary_flowgraph.cc#L891
+[`dart::InlineCacheMissHandler`]: https://github.com/dart-lang/sdk/blob/2ed6ea29003476e2a28fb5f4683a656427eb41ff/runtime/vm/runtime_entry.cc#L2493
+[`runtime/vm/compiler/compiler_pass.cc`]: https://github.com/dart-lang/sdk/blob/2ed6ea29003476e2a28fb5f4683a656427eb41ff/runtime/vm/compiler/compiler_pass.cc
+[`dart::JitCallSpecializer`]: https://github.com/dart-lang/sdk/blob/2ed6ea29003476e2a28fb5f4683a656427eb41ff/runtime/vm/compiler/jit/jit_call_specializer.h#L16
+[`pkg/vm/bin/gen_kernel.dart`]: https://github.com/dart-lang/sdk/blob/2ed6ea29003476e2a28fb5f4683a656427eb41ff/pkg/vm/bin/gen_kernel.dart
+[`pkg/vm/bin/dump_kernel.dart`]: https://github.com/dart-lang/sdk/blob/2ed6ea29003476e2a28fb5f4683a656427eb41ff/pkg/vm/bin/dump_kernel.dart
+[`pkg/front_end/tool/_fasta/compile_platform.dart`]: https://github.com/dart-lang/sdk/blob/2ed6ea29003476e2a28fb5f4683a656427eb41ff/pkg/front_end/tool/_fasta/compile_platform.dart
+[`package:kernel/ast.dart`]: https://github.com/dart-lang/sdk/blob/2ed6ea29003476e2a28fb5f4683a656427eb41ff/pkg/kernel/lib/ast.dart
+[`package:front_end`]: https://github.com/dart-lang/sdk/blob/2ed6ea29003476e2a28fb5f4683a656427eb41ff/pkg/front_end/lib
+[`dart::kernel::KernelLoader::LoadEntireProgram`]: https://github.com/dart-lang/sdk/blob/2ed6ea29003476e2a28fb5f4683a656427eb41ff/runtime/vm/kernel_loader.cc#L236
+[`pkg/vm/bin/kernel_service.dart`]: https://github.com/dart-lang/sdk/blob/2ed6ea29003476e2a28fb5f4683a656427eb41ff/pkg/vm/bin/kernel_service.dart
+[`runtime/vm/kernel_isolate.cc`]: https://github.com/dart-lang/sdk/blob/2ed6ea29003476e2a28fb5f4683a656427eb41ff/runtime/vm/kernel_isolate.cc
+[`package:vm`]: https://github.com/dart-lang/sdk/blob/2ed6ea29003476e2a28fb5f4683a656427eb41ff/pkg/vm/lib
+[`dart::Isolate`]: https://github.com/dart-lang/sdk/blob/2ed6ea29003476e2a28fb5f4683a656427eb41ff/runtime/vm/isolate.h#L971
+[`dart::IsolateGroup`]: https://github.com/dart-lang/sdk/blob/2ed6ea29003476e2a28fb5f4683a656427eb41ff/runtime/vm/isolate.h#L281
+[`dart::Heap`]: https://github.com/dart-lang/sdk/blob/2ed6ea29003476e2a28fb5f4683a656427eb41ff/runtime/vm/heap/heap.h#L35
+[`dart::Thread`]: https://github.com/dart-lang/sdk/blob/2ed6ea29003476e2a28fb5f4683a656427eb41ff/runtime/vm/thread.h#L334
+[`Dart_RunLoop`]: https://github.com/dart-lang/sdk/blob/2ed6ea29003476e2a28fb5f4683a656427eb41ff/runtime/vm/dart_api_impl.cc#L2010
+[`dart::MessageHandler`]: https://github.com/dart-lang/sdk/blob/2ed6ea29003476e2a28fb5f4683a656427eb41ff/runtime/vm/message_handler.h#L20
\ No newline at end of file
diff --git a/runtime/tools/wiki/CustomShellSessionPygmentsLexer/custom_shell_session/__init__.py b/runtime/tools/wiki/CustomShellSessionPygmentsLexer/custom_shell_session/__init__.py
deleted file mode 100644
index 58baf98..0000000
--- a/runtime/tools/wiki/CustomShellSessionPygmentsLexer/custom_shell_session/__init__.py
+++ /dev/null
@@ -1,5 +0,0 @@
-#
-# Copyright (c) 2020, the Dart project authors. Please see the AUTHORS file
-# for details. All rights reserved. Use of this source code is governed by a
-# BSD-style license that can be found in the LICENSE file.
-#
diff --git a/runtime/tools/wiki/CustomShellSessionPygmentsLexer/custom_shell_session/lexer.py b/runtime/tools/wiki/CustomShellSessionPygmentsLexer/custom_shell_session/lexer.py
deleted file mode 100644
index 3e1e635..0000000
--- a/runtime/tools/wiki/CustomShellSessionPygmentsLexer/custom_shell_session/lexer.py
+++ /dev/null
@@ -1,49 +0,0 @@
-#
-# Copyright (c) 2020, the Dart project authors. Please see the AUTHORS file
-# for details. All rights reserved. Use of this source code is governed by a
-# BSD-style license that can be found in the LICENSE file.
-#
-"""Simple lexer for shell sessions.
-
-Highlights command lines (lines starting with $ prompt) and comments starting
-with #. For example:
-
- # This is a comment
- $ this-is-a-command
- This is output of the command
-
- $ this-is-a-multiline-command with-some-arguments \
- and more arguments \
- and more arguments
- And some output.
-
-"""
-
-from pygments.lexer import RegexLexer, words
-from pygments.token import Comment, Generic, Keyword
-
-_comment_style = Comment
-# Note: there is a slight inversion of styles to make it easier to read.
-# We highlight output with Prompt style and command as a normal text.
-_output_style = Generic.Prompt
-_command_style = Generic.Text
-_prompt_style = Keyword
-
-
-class CustomShellSessionLexer(RegexLexer):
- name = 'CustomShellSession'
- aliases = ['custom-shell-session']
- filenames = ['*.log']
- tokens = {
- 'root': [
- (r'#.*\n', _comment_style),
- (r'^\$', _prompt_style, 'command'),
- (r'.', _output_style),
- ],
- 'command': [
- (r'\\\n', _command_style), # Continue in 'command' state.
- (r'$', _command_style, '#pop'), # End of line without escape.
- (r'.',
- _command_style), # Anything else continue in 'command' state.
- ]
- }
diff --git a/runtime/tools/wiki/CustomShellSessionPygmentsLexer/setup.py b/runtime/tools/wiki/CustomShellSessionPygmentsLexer/setup.py
deleted file mode 100644
index b00ade9..0000000
--- a/runtime/tools/wiki/CustomShellSessionPygmentsLexer/setup.py
+++ /dev/null
@@ -1,15 +0,0 @@
-#
-# Copyright (c) 2020, the Dart project authors. Please see the AUTHORS file
-# for details. All rights reserved. Use of this source code is governed by a
-# BSD-style license that can be found in the LICENSE file.
-#
-from setuptools import setup, find_packages
-
-setup(
- name='custom-shell-session',
- packages=find_packages(),
- entry_points="""
- [pygments.lexers]
- custom-shell-session = custom_shell_session.lexer:CustomShellSessionLexer
- """,
-)
diff --git a/runtime/tools/wiki/README.md b/runtime/tools/wiki/README.md
index 2606ac8..e8d5ede 100644
--- a/runtime/tools/wiki/README.md
+++ b/runtime/tools/wiki/README.md
@@ -10,33 +10,37 @@
# Markdown extensions
-## Asides
+## Admonitions and Asides
-Paragraphs wrapped into `<aside>...</aside>` will be rendered as a sidenote on
-margins of the page.
+Blockquotes starting with `> **Marker**` are converted either:
-## Cross-references `@{ref|text}`
+- into sidenotes (if `Marker` is `Note`), which will be rendered on margins
+of the page;
+- admonitions (if `Marker` is `Source to read`, `Trying it` or `Warning`).
-Cross-references are rendered as links to GitHub at the current commit.
+## Referencing C++ symbols and files
-* `@{file-path}` is just rendered a link to the given file;
-* `@{package:name/path.dart}` is rendered as a link to file `path.dart` within
-package `name` - actual path is resolved via root `.packages` file in the SDK
-root;
-* `@{c++-symbol}` is rendered as a link to the line in the file which defines
+Script extends Markdown references with special support for references that
+use ``[`ref`][]`` and ``[text][`ref`]``. The following values for `ref` are
+recognized and resolved as links to GitHub at the current commit.
+
+* `file-path` is resolved as a link to the given file;
+* `package:name/path.dart` is resolved as a link to file `path.dart` within
+package `name` - actual path is resolved via `.dart_tool/package_config.json`
+file in the SDK root;
+* `c++ symbol` is resolved as a link to the line in the file which defines
the given C++ symbol.
+If markdown file contains any references in this form then running
+`runtime/tools/wiki/build/build.py --deploy` will generate a reference
+section at the end of the file. Appending this section allows other Markdown
+tools (e.g. GitHub viewer) to render such special links correctly.
+
# Prerequisites
1. Install all Python dependencies.
```console
$ pip3 install coloredlogs jinja2 markdown aiohttp watchdog pymdown-extensions pygments
```
-2. Install the custom pygments lexer we use for shell session examples:
- ```
- $ cd runtime/tools/wiki/CustomShellSessionPygmentsLexer
- $ python3 setup.py develop
- ```
+2. Install `libclang` (`brew install llvm` on Mac OS X).
3. Install SASS compiler (make sure that SASS binary is in your path).
-4. Generate `xref.json` file following instructions in
-`xref_extractor/README.md`.
\ No newline at end of file
diff --git a/runtime/tools/wiki/build/admonitions.py b/runtime/tools/wiki/build/admonitions.py
new file mode 100644
index 0000000..18f61ab
--- /dev/null
+++ b/runtime/tools/wiki/build/admonitions.py
@@ -0,0 +1,61 @@
+# Copyright (c) 2023, the Dart project authors. Please see the AUTHORS file
+# for details. All rights reserved. Use of this source code is governed by a
+# BSD-style license that can be found in the LICENSE file.
+#
+"""Preprocesses Markdown source and converts admonitions written in blockquotes.
+
+Usage: convert_admonitions(str)
+"""
+
+import re
+
+_RECOGNIZED_ADMONITIONS = {
+ "Source to read": "sourcecode",
+ "Trying it": "tryit",
+ "Warning": "warning"
+}
+
+
+def convert_admonitions(content: str) -> str:
+ """Convert blockquotes into admonitions if they start with a marker.
+
+ Blockquotes starting with `> **marker**` are converted either into
+ sidenotes (`<span class="aside"/>`) or into admonitions to be
+ processed by an admonition extension later.
+ """
+ processed = []
+ current_admonition = None
+ indent = ''
+ for line in content.split('\n'):
+ if current_admonition is not None:
+ if line.startswith('>'):
+ processed.append(indent + line[1:])
+ continue
+ if current_admonition == 'Note':
+ note = processed.pop()
+ processed.pop()
+ processed[-1] = processed[
+ -1] + f' <span class="aside" markdown=1>{note}</span>'
+ current_admonition = None
+ elif line.startswith('> **') and line.endswith('**'):
+ current_admonition = re.match(r'^> \*\*(.*)\*\*$', line)[1]
+ if current_admonition == 'Note':
+ indent = ''
+
+ # Drop all empy lines preceeding the side note.
+ while processed[-1] == '':
+ processed.pop()
+
+ # Do not try to attach sidenote to the section title.
+ if processed[-1].startswith('#'):
+ processed.append('')
+ else:
+ # Start an admonition using Python markdown syntax.
+ processed.append(
+ f'!!! {_RECOGNIZED_ADMONITIONS[current_admonition]} "{current_admonition}"'
+ )
+ current_admonition = True
+ indent = ' '
+ continue
+ processed.append(line)
+ return "\n".join(processed)
diff --git a/runtime/tools/wiki/build/build.py b/runtime/tools/wiki/build/build.py
index 6fc31dd..7fe75dc 100755
--- a/runtime/tools/wiki/build/build.py
+++ b/runtime/tools/wiki/build/build.py
@@ -17,35 +17,32 @@
from __future__ import annotations
+import logging
+import os
+import re
+import shutil
+import subprocess
+from pathlib import Path
+from typing import Callable, Dict, Sequence
+
import argparse
import asyncio
import codecs
import coloredlogs
-import glob
import jinja2
-import logging
import markdown
-import os
-import posixpath
-import re
-import shutil
-import subprocess
-import sys
-import time
-import urllib
from aiohttp import web, WSCloseCode, WSMsgType
-from http.server import HTTPServer, SimpleHTTPRequestHandler
from markdown.extensions.codehilite import CodeHiliteExtension
-from pathlib import Path
-from typing import Callable, Dict, Sequence
from watchdog.events import FileSystemEventHandler
from watchdog.observers import Observer
from xrefs import XrefExtension
+from admonitions import convert_admonitions
# Configure logging to use colors.
-coloredlogs.install(
- level='INFO', fmt='%(asctime)s - %(message)s', datefmt='%H:%M:%S')
+coloredlogs.install(level='INFO',
+ fmt='%(asctime)s - %(message)s',
+ datefmt='%H:%M:%S')
# Declare various directory paths.
# We expected to be located in runtime/tools/wiki/build.
@@ -73,16 +70,17 @@
# Parse incoming arguments.
parser = argparse.ArgumentParser()
parser.add_argument('--deploy', dest='deploy', action='store_true')
+parser.add_argument('--deployment-root', dest='deployment_root', default='')
parser.set_defaults(deploy=False)
args = parser.parse_args()
is_dev_mode = not args.deploy
+deployment_root = args.deployment_root
# Initialize jinja environment.
-jinja2_env = jinja2.Environment(
- loader=jinja2.FileSystemLoader(TEMPLATES_DIR),
- lstrip_blocks=True,
- trim_blocks=True)
+jinja2_env = jinja2.Environment(loader=jinja2.FileSystemLoader(TEMPLATES_DIR),
+ lstrip_blocks=True,
+ trim_blocks=True)
class Artifact:
@@ -92,19 +90,19 @@
all: Dict[str, Artifact] = {}
# List of listeners which are notified whenever some artifact is rebuilt.
- listeners = []
+ listeners: list[Callable] = []
def __init__(self, output: str, inputs: Sequence[str]):
Artifact.all[output] = self
self.output = output
- self.inputs = inputs
+ self.inputs = [os.path.normpath(input) for input in inputs]
def depends_on(self, path: str) -> bool:
"""Check if this"""
return path in self.inputs
def build(self):
- pass
+ """Convert artifact inputs into an output."""
@staticmethod
def build_all():
@@ -112,11 +110,11 @@
Artifact.build_matching(lambda obj: True)
@staticmethod
- def build_matching(filter: Callable[[Artifact], bool]):
+ def build_matching(predicate: Callable[[Artifact], bool]):
"""Build all artifacts matching the given filter."""
rebuilt = False
for _, artifact in Artifact.all.items():
- if filter(artifact):
+ if predicate(artifact):
artifact.build()
rebuilt = True
@@ -126,46 +124,74 @@
listener()
+xref_extension = XrefExtension()
+
+
class Page(Artifact):
"""A single wiki Page (a markdown file)."""
def __init__(self, name: str):
self.name = name
- super().__init__(
- os.path.join(OUTPUT_DIR, name + '.html'),
- [os.path.join(WIKI_SOURCE_DIR, name + '.md')])
+ super().__init__(os.path.join(OUTPUT_DIR, name + '.html'),
+ [os.path.join(WIKI_SOURCE_DIR, name + '.md')])
def __repr__(self):
- return 'Page(%s <- %s)' % (self.output, self.inputs[0])
+ return f'Page({self.output} <- {self.inputs[0]})'
def depends_on(self, path: str):
return path.startswith(TEMPLATES_INCLUDES_DIR) or super().depends_on(
path)
- def load_markdown(self):
- with open(self.inputs[0], 'r') as file:
+ def _load_markdown(self):
+ with open(self.inputs[0], 'r', encoding='utf-8') as file:
content = file.read()
- content = re.sub(r'(?<=[^\n])\n+<aside>', '<span class="aside">',
- content)
- content = re.sub(r'</aside>', '</span>', content)
- return content
+
+ # Remove autogenerated xref section.
+ content = re.sub(r'<!\-\- AUTOGENERATED XREF SECTION \-\->.*$',
+ '',
+ content,
+ flags=re.DOTALL)
+
+ return convert_admonitions(content)
+
+ def _update_xref_section(self, xrefs):
+ with open(self.inputs[0], 'r', encoding='utf-8') as file:
+ content = file.read()
+ section = '\n'.join(
+ ['', '<!-- AUTOGENERATED XREF SECTION -->'] +
+ [f'[{key}]: {value}' for key, value in xrefs.items()])
+ with open(self.inputs[0], 'w', encoding='utf-8') as file:
+ content = re.sub(r'\n<!-- AUTOGENERATED XREF SECTION -->.*$',
+ '',
+ content,
+ flags=re.DOTALL)
+ content += section
+ file.write(content)
def build(self):
logging.info('Build %s from %s', self.output, self.inputs[0])
template = jinja2_env.get_template(PAGE_TEMPLATE)
+ md_converter = markdown.Markdown(extensions=[
+ 'admonition',
+ 'extra',
+ CodeHiliteExtension(),
+ 'tables',
+ 'pymdownx.superfences',
+ 'toc',
+ xref_extension,
+ ])
result = template.render({
'dev':
is_dev_mode,
'body':
- markdown.markdown(
- self.load_markdown(),
- extensions=[
- 'admonition', 'extra',
- CodeHiliteExtension(), 'tables', 'pymdownx.superfences',
- XrefExtension()
- ])
+ md_converter.convert(self._load_markdown()),
+ 'root':
+ deployment_root
})
+ # pylint: disable=no-member
+ if not is_dev_mode and len(md_converter.xrefs) > 0:
+ self._update_xref_section(md_converter.xrefs)
os.makedirs(os.path.dirname(self.output), exist_ok=True)
with codecs.open(self.output, "w", encoding='utf-8') as file:
@@ -180,12 +206,11 @@
def __init__(self, name: str):
self.name = name
- super().__init__(
- os.path.join(OUTPUT_CSS_DIR, name + '.css'),
- [os.path.join(STYLES_DIR, name + '.scss')])
+ super().__init__(os.path.join(OUTPUT_CSS_DIR, name + '.css'),
+ [os.path.join(STYLES_DIR, name + '.scss')])
def __repr__(self):
- return 'Style(%s <- %s)' % (self.output, self.inputs[0])
+ return f'Style({self.output} <- {self.inputs[0]})'
def depends_on(self, path: str):
return path.startswith(STYLES_INCLUDES_DIR) or super().depends_on(path)
@@ -206,15 +231,17 @@
def find_artifacts():
"""Find all wiki pages and styles and create corresponding Artifacts."""
Artifact.all = {}
- for f in Path(WIKI_SOURCE_DIR).rglob('*.md'):
- name = f.relative_to(Path(WIKI_SOURCE_DIR)).as_posix().rsplit('.', 1)[0]
+ for file in Path(WIKI_SOURCE_DIR).rglob('*.md'):
+ name = file.relative_to(Path(WIKI_SOURCE_DIR)).as_posix().rsplit(
+ '.', 1)[0]
Page(name)
- for f in Path(STYLES_DIR).glob('*.scss'):
- Style(f.stem)
+ for file in Path(STYLES_DIR).glob('*.scss'):
+ Style(file.stem)
def build_for_deploy():
+ """Create a directory which can be deployed to static hosting."""
logging.info('Building wiki for deployment into %s', OUTPUT_DIR)
Artifact.build_all()
for images_dir in find_images_directories():
@@ -234,15 +261,13 @@
class ArtifactEventHandler(FileSystemEventHandler):
"""File system listener rebuilding artifacts based on changed paths."""
- def __init__(self):
- super().__init__()
-
def on_modified(self, event):
- Artifact.build_matching(
- lambda artifact: artifact.depends_on(event.src_path))
+ path = os.path.relpath(event.src_path, '.')
+ Artifact.build_matching(lambda artifact: artifact.depends_on(path))
def serve_for_development():
+ """Serve wiki for development (with hot refresh)."""
logging.info('Serving wiki for development')
Artifact.build_all()
@@ -257,8 +282,8 @@
async def on_shutdown(app):
for ws in app['websockets']:
- await ws.close(
- code=WSCloseCode.GOING_AWAY, message='Server shutdown')
+ await ws.close(code=WSCloseCode.GOING_AWAY,
+ message='Server shutdown')
observer.stop()
observer.join()
@@ -320,6 +345,7 @@
def main():
+ """Main entry point."""
find_artifacts()
if is_dev_mode:
serve_for_development()
diff --git a/runtime/tools/wiki/build/cpp_indexer.py b/runtime/tools/wiki/build/cpp_indexer.py
new file mode 100644
index 0000000..489cd53
--- /dev/null
+++ b/runtime/tools/wiki/build/cpp_indexer.py
@@ -0,0 +1,295 @@
+# Copyright (c) 2023, the Dart project authors. Please see the AUTHORS file
+# for details. All rights reserved. Use of this source code is governed by a
+# BSD-style license that can be found in the LICENSE file.
+#
+"""Clang based C++ code indexer which produces xref.json."""
+from __future__ import annotations
+
+import glob
+import json
+import logging
+import os
+import platform
+import re
+import sys
+import subprocess
+
+from contextlib import contextmanager
+from dataclasses import dataclass, field
+from typing import Dict, Optional, Set
+
+from marshmallow_dataclass import class_schema
+from progress.bar import ShadyBar
+
+_libclang_path = None
+_clang_include_dir = None
+if platform.system() == 'Darwin':
+ _path_to_llvm = subprocess.check_output(['brew', '--prefix', 'llvm'],
+ encoding='utf-8').strip()
+ _clang_include_dir = glob.glob(f'{_path_to_llvm}/lib/clang/**/include')[0]
+ _libclang_path = f'{_path_to_llvm}/lib/libclang.dylib'
+ sys.path.append(
+ f'{_path_to_llvm}/lib/python{sys.version_info.major}.{sys.version_info.minor}/site-packages'
+ )
+
+# pylint: disable=wrong-import-position,line-too-long
+from clang.cindex import Config, CompilationDatabase, Cursor, CursorKind, Index, SourceLocation, TranslationUnit
+
+if _libclang_path is not None:
+ Config.set_library_file(_libclang_path)
+
+_DART_CONFIGURATION = 'Release' + (
+ 'ARM64' if platform.uname().machine == 'arm64' else 'X64')
+_DART_BUILD_DIR = ('xcodebuild' if platform.system() == 'Darwin' else
+ 'out') + '/' + _DART_CONFIGURATION
+
+
+@contextmanager
+def _change_working_directory_to(path):
+ oldpwd = os.getcwd()
+ os.chdir(path)
+ try:
+ yield
+ finally:
+ os.chdir(oldpwd)
+
+
+def _create_compilation_database():
+ """Create compilation database for the default configuration.
+
+ Extacts compilation database (compile_commands.json) for the default
+ build configuration and filters it down to commands taht builds just
+ the core VM pieces in the host configuration.
+ """
+ logging.info('Extracting compilation commands from build files for %s',
+ _DART_CONFIGURATION)
+ with _change_working_directory_to(_DART_BUILD_DIR):
+ commands = json.loads(
+ subprocess.check_output(['ninja', '-C', '.', '-t', 'compdb',
+ 'cxx']))
+ pattern = re.compile(
+ r'libdart(_vm|_compiler)?_precompiler_host_targeting_host\.')
+ with open('compile_commands.json', 'w', encoding='utf-8') as outfile:
+ json.dump([
+ cmd for cmd in commands
+ if pattern.search(cmd['command']) is not None
+ ], outfile)
+
+
+def _get_current_commit_hash():
+ return subprocess.check_output(['git', 'merge-base', 'main', 'HEAD'],
+ text=True).strip()
+
+
+@dataclass
+class _ClassInfo:
+ location: str
+ members: Dict[str, str] = field(default_factory=dict)
+
+
+@dataclass
+class Location:
+ """Symbol location referring to a line in a specific file."""
+ filename: str
+ lineno: int
+
+
+@dataclass
+class SymbolsIndex:
+ """Index of C++ symbols extracted from source."""
+ commit: str
+ files: list[str] = field(default_factory=list)
+ classes: Dict[str, _ClassInfo] = field(default_factory=dict)
+ functions: Dict[str, str] = field(default_factory=dict)
+
+ def try_resolve(self, ref: str) -> Optional[Location]:
+ """Resolve the location of the given reference."""
+ loc = self._try_resolve_impl(ref)
+ if loc is not None:
+ return self._location_from_string(loc)
+ return None
+
+ def _try_resolve_impl(self, ref: str) -> Optional[str]:
+ if ref in self.functions:
+ return self.functions[ref]
+ if ref in self.classes:
+ return self.classes[ref].location
+ if '::' in ref:
+ (class_name, function_name) = ref.rsplit('::', 1)
+ if class_name in self.classes:
+ return self.classes[class_name].members.get(function_name)
+ return None
+
+ def _location_from_string(self, loc: str) -> Location:
+ (file_idx, line_idx) = loc.split(':', 1)
+ return Location(self.files[int(file_idx)], int(line_idx))
+
+
+class _Indexer:
+ symbols_index: SymbolsIndex
+
+ processed_files: Set[str]
+ classes_by_usr: Dict[str, _ClassInfo]
+
+ name_stack: list[str]
+ info_stack: list[_ClassInfo]
+ files_seen_in_unit: Set[str]
+
+ def __init__(self):
+ self.symbols_index = SymbolsIndex(commit=_get_current_commit_hash())
+
+ self.processed_files = set()
+ self.classes_by_usr = {}
+ self.files_index = {}
+
+ self.name_stack = []
+ self.info_stack = []
+ self.files_seen_in_unit = set()
+
+ def index(self, unit: TranslationUnit):
+ """Index the given translation unit and append new symbols to index."""
+ self.name_stack.clear()
+ self.info_stack.clear()
+ self.files_seen_in_unit.clear()
+ self._recurse(unit.cursor)
+ self.processed_files |= self.files_seen_in_unit
+
+ def _recurse(self, cursor: Cursor):
+ name = ""
+ kind = cursor.kind
+
+ if cursor.location.file is not None:
+ name = cursor.location.file.name
+ if name in self.processed_files or not name.startswith('../..'):
+ return
+ self.files_seen_in_unit.add(name)
+
+ if kind == CursorKind.CLASS_DECL:
+ if not cursor.is_definition():
+ return
+ usr = cursor.get_usr()
+ if usr in self.classes_by_usr:
+ return
+ self.name_stack.append(cursor.spelling)
+ class_name = '::'.join(self.name_stack)
+ class_info = _ClassInfo(self._format_location(cursor.location))
+ self.info_stack.append(class_info)
+ self.symbols_index.classes[class_name] = class_info
+ self.classes_by_usr[usr] = class_info
+ elif kind == CursorKind.NAMESPACE:
+ self.name_stack.append(cursor.spelling)
+ elif kind == CursorKind.FUNCTION_DECL and cursor.is_definition():
+ namespace_prefix = ""
+ if cursor.semantic_parent.kind == CursorKind.NAMESPACE:
+ namespace_prefix = '::'.join(self.name_stack) + (
+ '::' if len(self.name_stack) > 0 else "")
+ function_name = namespace_prefix + cursor.spelling
+ self.symbols_index.functions[function_name] = self._format_location(
+ cursor.location)
+ return
+ elif kind == CursorKind.CXX_METHOD and cursor.is_definition():
+ parent = cursor.semantic_parent
+ if parent.kind == CursorKind.CLASS_DECL:
+ class_info_or_none = self.classes_by_usr.get(parent.get_usr())
+ if class_info_or_none is None:
+ return
+ class_info_or_none.members[
+ cursor.spelling] = self._format_location(cursor.location)
+ return
+ elif kind == CursorKind.VAR_DECL and cursor.is_definition():
+ parent = cursor.semantic_parent
+ if parent.kind == CursorKind.CLASS_DECL:
+ class_info_or_none = self.classes_by_usr.get(parent.get_usr())
+ if class_info_or_none is None:
+ return
+ class_info_or_none.members[
+ cursor.spelling] = self._format_location(cursor.location)
+
+ for child in cursor.get_children():
+ self._recurse(child)
+
+ if kind == CursorKind.NAMESPACE:
+ self.name_stack.pop()
+ elif kind == CursorKind.CLASS_DECL:
+ self.name_stack.pop()
+ self.info_stack.pop()
+
+ def _format_location(self, loc: SourceLocation):
+ file_name = loc.file.name
+ lineno = loc.line
+ return f'{self._get_file_index(file_name)}:{lineno}'
+
+ def _get_file_index(self, file_name: str):
+ index = self.files_index.get(file_name)
+ if index is None:
+ index = len(self.symbols_index.files)
+ self.files_index[file_name] = index
+ self.symbols_index.files.append(
+ os.path.relpath(os.path.abspath(file_name),
+ os.path.abspath('../..')))
+ return index
+
+
+def _index_source() -> SymbolsIndex:
+ indexer = _Indexer()
+
+ _create_compilation_database()
+ with _change_working_directory_to(_DART_BUILD_DIR):
+ index = Index.create()
+ compdb = CompilationDatabase.fromDirectory('.')
+
+ commands = list(compdb.getAllCompileCommands())
+ with ShadyBar('Indexing',
+ max=len(commands),
+ suffix='%(percent)d%% eta %(eta_td)s') as progress_bar:
+ for command in commands:
+ args = [
+ arg for arg in command.arguments
+ if arg.startswith('-I') or arg.startswith('-W') or
+ arg.startswith('-D') or arg.startswith('-i') or
+ arg.startswith('sdk/') or arg.startswith('-std')
+ ] + [
+ '-Wno-macro-redefined', '-Wno-unused-const-variable',
+ '-Wno-unused-function', '-Wno-unused-variable'
+ ]
+
+ if _clang_include_dir is not None:
+ args.append(f'-I{_clang_include_dir}')
+
+ unit = index.parse(command.filename, args=args)
+ for diag in unit.diagnostics:
+ print(diag.format())
+
+ indexer.index(unit)
+ progress_bar.next()
+
+ return indexer.symbols_index
+
+
+_SymbolsIndexSchema = class_schema(SymbolsIndex)()
+
+
+def load_index(filename: str) -> SymbolsIndex:
+ """Load symbols index from the given file.
+
+ If index is out of date or missing it will be generated.
+ """
+ index: SymbolsIndex
+
+ if os.path.exists(filename):
+ with open(filename, 'r', encoding='utf-8') as json_file:
+ index = _SymbolsIndexSchema.loads(json_file.read())
+ if _get_current_commit_hash() == index.commit:
+ logging.info('Loaded symbols index from %s', filename)
+ return index
+ logging.warning(
+ '%s is generated for commit %s while current commit is %s',
+ filename, index.commit, _get_current_commit_hash())
+
+ index = _index_source()
+ with open(filename, 'w', encoding='utf-8') as json_file:
+ json_file.write(_SymbolsIndexSchema.dumps(index))
+ logging.info(
+ 'Successfully indexed C++ source and written symbols index into %s',
+ filename)
+ return index
diff --git a/runtime/tools/wiki/build/xrefs.py b/runtime/tools/wiki/build/xrefs.py
index bb8a1b1..c351340 100644
--- a/runtime/tools/wiki/build/xrefs.py
+++ b/runtime/tools/wiki/build/xrefs.py
@@ -2,119 +2,158 @@
# for details. All rights reserved. Use of this source code is governed by a
# BSD-style license that can be found in the LICENSE file.
#
-"""Support for @{ref|text} reference links in Markdown.
+"""Markdown extension for xrefs and reference to other Markdown files.
-This Markdown extension converts @{ref|text} into a link with the given
-[text] pointing to a particular source code location. [ref] can be one of
-the following:
+Xref is a reference of form [`symbol`][] or [text][`symbol`], where symbol
+is expected to be one of the following:
* package:-scheme URI - it will be resolved using .packages file in the
root directory
* file path
* C++ symbol - will be resolved through xref.json file (see README.md)
+Xrefs are converted to GitHub links.
+
+Additionally this extension retargets links pointing to markdown files to
+the html files produced from these markdown files.
+
Usage: markdown.markdown(extensions=[XrefExtension()])
"""
import json
import logging
import os
-import subprocess
+import re
-from markdown.extensions import Extension
-from markdown.inlinepatterns import InlineProcessor
-from markdown.util import etree
-from typing import Optional
+import xml.etree.ElementTree as etree
+from typing import Dict, Optional
from urllib.parse import urlparse
-_current_commit_hash = subprocess.run(['git', 'rev-parse', 'HEAD'],
- capture_output=True,
- encoding='utf8').stdout
-
-# Load .packages file into a dictionary.
-with open('.packages') as packages_file:
- _packages = dict([
- package_mapping.split(':', 1)
- for package_mapping in packages_file
- if not package_mapping.startswith('#')
- ])
-
-# Load xref.json and verify that it was generated for the current commit to
-# avoid discrepancies in the generated links.
-with open('xref.json') as json_file:
- _xrefs = json.load(json_file)
-
-if _current_commit_hash != _xrefs['commit']:
- logging.error(
- 'xref.json is generated for commit %s while current commit is %s',
- _xrefs['commit'], _current_commit_hash)
-
-
-def _make_github_uri(file: str, lineno: str = None) -> str:
- """Generates source link pointing to GitHub"""
- fragment = '#L%s' % (lineno) if lineno is not None else ''
- return 'https://github.com/dart-lang/sdk/blob/%s/%s%s' % (
- _current_commit_hash, file, fragment)
-
-
-def _file_ref_to_github_uri(file_ref: str) -> str:
- """Generates source link pointing to GitHub from an xref.json reference."""
- (file_idx, line_idx) = file_ref.split(':', 1)
- return _make_github_uri(_xrefs['files'][int(file_idx)], line_idx)
-
-
-def _resolve_ref_via_xref(ref: str) -> Optional[str]:
- """Resolve the target of the given reference via xref.json"""
- if ref in _xrefs['functions']:
- return _xrefs['functions'][ref]
- if ref in _xrefs['classes']:
- return _xrefs['classes'][ref][0]
- if '::' in ref:
- (class_name, function_name) = ref.rsplit('::', 1)
- if class_name in _xrefs['classes'] and len(
- _xrefs['classes'][class_name]) == 2:
- return _xrefs['classes'][class_name][1][function_name]
- logging.error('Failed to resolve xref %s' % ref)
- return None
-
-
-def _resolve_ref(ref: str) -> Optional[str]:
- if ref.startswith('package:'):
- # Resolve as package uri via .packages.
- uri = urlparse(ref)
- (package_name, *path_to_file) = uri.path.split('/', 1)
- package_path = _packages[package_name]
- if len(path_to_file) == 0:
- return _make_github_uri(package_path)
- else:
- return _make_github_uri(os.path.join(package_path, path_to_file[0]))
- elif os.path.exists(ref):
- # Resolve as a file link.
- return _make_github_uri(_current_commit_hash, ref)
- else:
- # Resolve as a C++ symbol via xref.json
- file_ref = _resolve_ref_via_xref(ref)
- if file_ref is not None:
- return _file_ref_to_github_uri(file_ref)
+from cpp_indexer import SymbolsIndex, load_index
+from markdown.extensions import Extension
+from markdown.inlinepatterns import InlineProcessor
+from markdown.treeprocessors import Treeprocessor
class _XrefPattern(InlineProcessor):
- """InlineProcessor responsible for handling @{ref|text} syntax."""
+ """Converts xrefs into GitHub links.
+
+ Recognizes [`symbol`][] and [text][`symbol`] link formats where symbol
+ is expected to be one of the following:
+
+ * Fully qualified reference to a C++ class, method or function;
+ * Package URI pointing to one of the packages included in the SDK
+ checkout.
+ * File reference to one of the file in the SDK.
+ """
+
+ XREF_RE = r'\[`(?P<symbol>[^]]+)`\]?\[\]|\[(?P<text>[^]]*)\]\[`(?P<target>[^]]+)`\]'
+
+ def __init__(self, md, symbols_index: SymbolsIndex,
+ packages: Dict[str, str]):
+ super().__init__(_XrefPattern.XREF_RE)
+ self.symbols_index = symbols_index
+ self.packages = packages
+ self.md = md
def handleMatch(self, m, data):
- ref = m.group(1)
- text = m.group(2)
- uri = _resolve_ref(ref)
- el = etree.Element('a')
- el.attrib['href'] = uri
- el.attrib['target'] = 'blank'
- el.text = text[1:] if text is not None else ref
- return el, m.start(0), m.end(0)
+ text = m.group('text')
+ symbol = m.group('symbol')
+ if symbol is None:
+ symbol = m.group('target')
+
+ uri = self._resolve_ref(symbol) or '#broken-link'
+
+ # Remember this xref. build process can later use this information
+ # to produce xref reference section at the end of the markdown file.
+ self.md.xrefs[f"`{symbol}`"] = uri
+
+ # Create <a href='uri'>text</a> element. If text is not defined
+ # simply use a slightly sanitized symbol name.
+ anchor = etree.Element('a')
+ anchor.attrib['href'] = uri
+ anchor.attrib['target'] = '_blank'
+ if text is not None:
+ anchor.text = text
+ else:
+ code = etree.Element('code')
+ code.text = re.sub(r'^dart::', '', symbol)
+ anchor.append(code)
+
+ # Replace the whole pattern match with anchor element.
+ return anchor, m.start(0), m.end(0)
+
+ def _resolve_ref(self, ref: str) -> Optional[str]:
+ if ref.startswith('package:'):
+ # Resolve as package uri via .packages.
+ uri = urlparse(ref)
+ (package_name, *path_to_file) = uri.path.split('/', 1)
+ package_path = self.packages[package_name]
+ if len(path_to_file) == 0:
+ return self._make_github_uri(package_path)
+ else:
+ return self._make_github_uri(
+ os.path.join(package_path, path_to_file[0]))
+ elif os.path.exists(ref):
+ # Resolve as a file link.
+ return self._make_github_uri(ref)
+ else:
+ # Resolve as a symbol.
+ loc = self.symbols_index.try_resolve(ref)
+ if loc is not None:
+ return self._make_github_uri(loc.filename, loc.lineno)
+
+ logging.error('Failed to resolve xref %s', ref)
+ return None
+
+ def _make_github_uri(self, file: str, lineno: Optional[int] = None) -> str:
+ """Generates source link pointing to GitHub"""
+ fragment = f'#L{lineno}' if lineno is not None else ''
+ return f'https://github.com/dart-lang/sdk/blob/{self.symbols_index.commit}/{file}{fragment}'
+
+
+class _MdLinkFixerTreeprocessor(Treeprocessor):
+ """Redirects links pointing to .md files to .html files built from them."""
+
+ def run(self, root):
+ for elem in root.iter('a'):
+ href = elem.get('href')
+ if href is None:
+ continue
+ parsed_href = urlparse(href)
+ if parsed_href.path.endswith('.md'):
+ elem.set(
+ 'href',
+ parsed_href._replace(path=parsed_href.path[:-3] +
+ '.html').geturl())
class XrefExtension(Extension):
- """Markdown extension responsible for expanding @{ref|text} into links."""
+ """Markdown extension which handles xrefs and links to markdown files."""
+ symbols_index: SymbolsIndex
+ packages: Dict[str, str]
+
+ def __init__(self) -> None:
+ super().__init__()
+ self.symbols_index = load_index('xref.json')
+ self.packages = XrefExtension._load_package_config()
def extendMarkdown(self, md):
+ md.xrefs = {}
+ md.treeprocessors.register(_MdLinkFixerTreeprocessor(), 'mdlinkfixer',
+ 0)
md.inlinePatterns.register(
- _XrefPattern(r'@{([^}|]*)(\|[^}]+)?}'), 'xref', 175)
+ _XrefPattern(md, self.symbols_index, self.packages), 'xref', 200)
+
+ @staticmethod
+ def _load_package_config() -> Dict[str, str]:
+ # Load package_config.json file into a dictionary.
+ with open('.dart_tool/package_config.json',
+ encoding='utf-8') as package_config_file:
+ package_config = json.load(package_config_file)
+ return dict([(pkg['name'],
+ os.path.normpath(
+ os.path.join('.dart_tool/', pkg['rootUri'],
+ pkg['packageUri'])))
+ for pkg in package_config['packages']
+ if 'packageUri' in pkg])
diff --git a/runtime/tools/wiki/styles/style.scss b/runtime/tools/wiki/styles/style.scss
index adfc9a5..5f4435c 100644
--- a/runtime/tools/wiki/styles/style.scss
+++ b/runtime/tools/wiki/styles/style.scss
@@ -1,7 +1,26 @@
-@import url('https://fonts.googleapis.com/css?family=Source+Code+Pro|Source+Sans+Pro&display=swap');
+@import url('https://rsms.me/inter/inter.css');
+@import url('https://rsms.me/inter/inter-display.css');
-$monospace: 'Source Code Pro', Menlo, Consolas, monospace;
-$sans: 'Source Sans Pro', sans-serif;
+@font-face {
+ font-named-instance:"Regular";font-family: jbmono;
+ font-style: normal;
+ font-weight: 100 800;
+ src: url(https://rsms.me/res/fonts/jbm/jetbrains-mono-wght.woff2) format("woff2")
+}
+
+@font-face {
+ font-named-instance:"Italic";font-family: jbmono;
+ font-style: italic;
+ font-weight: 100 800;
+ src: url(https://rsms.me/res/fonts/jbm/jetbrains-mono-italic_wght.woff2) format("woff2")
+}
+
+:root { font-family: 'Inter', sans-serif; }
+@supports (font-variation-settings: normal) {
+ :root { font-family: 'Inter var', sans-serif; font-weight: 460;}
+}
+
+$monospace: jbmono, 'IBM Plex Mono', Menlo, Consolas, monospace;
a {
color: #C00;
@@ -27,7 +46,7 @@
$code-line-height: 24px;
$header-font-size: 16px;
-div.codehilite {
+div.highlight {
font-size: $header-font-size;
margin: 0px -20px;
padding: 5px 20px;
@@ -48,8 +67,9 @@
margin: 0;
padding: 0;
height: 100%;
- font: normal #{$base-font-size}/#{$base-line-height} $sans;
- text-align: justify;
+ font-size: $base-font-size;
+ line-height: $base-line-height;
+ text-align: left;
color: #112;
text-rendering: optimizeLegibility;
-webkit-font-smoothing: antialiased;
@@ -57,7 +77,6 @@
}
h1, h2, h3, h4, h5 {
- font-family: $sans;
text-align: left;
position: relative;
}
@@ -151,45 +170,28 @@
position: relative;
right: 0px;
width: auto;
- text-align: justify;
- font-size: .9em;
- background: #F0F4C3;
+ text-align: left;
+ font-size: .8em;
+ line-height: 1.5em;
+ background-color: rgba(128,128,128,0.1);
display: block;
- padding: 5px;
+ padding: .8em;
}
body {
- font: normal 14px/18px $sans;
+ font-size: 14px;
+ line-height: 18px;
padding: 5px;
}
- div.codehilite {
- font: normal 10px/13px $monospace;
+ div.highlight {
+ font-size: 10px;
+ line-height: 13px;
margin: 0px -5px;
padding: 5px 5px;
}
}
-.book-title {
- font: 3em $monospace;
- text-transform: uppercase;
- font-weight: normal;
- /* border-width: 0px 1px 0px 1px; */
- /* border: dashed #C00; */
- border-width: 2px 0px 2px;
- padding: 10px 0px 5px 0px;
- text-align: center;
- letter-spacing: .1em;
- color: #C00;
- /* background-color: white; */
-}
-
-.book-subtitle {
- font-family: $monospace;
- text-align: center;
- letter-spacing: .1em;
-}
-
code {
font: normal #{$code-font-size}/#{$code-line-height} $monospace;
white-space: pre;
@@ -211,7 +213,6 @@
}
.start-reading span {
- font-family: $sans;
color: white;
background-color: #C00;
border-radius: 10px;
@@ -224,82 +225,125 @@
}
// Pygments
-.codehilite {
- background-color: #fdf6e3; color: #586e75;
- .hll {
- opacity: .4;
+.highlight {
+ line-height: initial;
+
+ pre {
+ --bg: #fff;
+ --fg: #000;
+ --comment: #6a737d;
+ --punctuation: inherit;
+ --operator: inherit;
+ --entity: #6f42c1;
+ --keyword: #d73a49;
+ --variable: #e36209;
+ --string-constant: #032f62; /* strings etc */
+ --numeric-constant: #005cc5;
+
+ padding: 1em;
+ background: var(--bg);
+ color: var(--fg);
+
+ /* punctuation */
+ & .p { color: var(--punctuation); }
+
+ /* syntax error */
+ /*& .err { color: #ff0015; }*/
+
+ /* names / identifiers */
+ /*& .n ,*/ /* Name */
+ & .na, /* Name.Attribute */
+ & .nb, /* Name.Builtin */
+ & .nc, /* Name.Class */
+ & .no, /* Name.Constant */
+ & .nd, /* Name.Decorator */
+ & .ni, /* Name.Entity */
+ & .ne, /* Name.Exception */
+ & .nf, /* Name.Function */
+ & .nl, /* Name.Label */
+ & .nn, /* Name.Namespace */
+ & .nx,
+ & .py,
+ & .nt /* Name.Tag */ {
+ color: var(--entity);
+ }
+
+ & .vc,
+ & .vg,
+ & .vi,
+ & .nv {
+ color: var(--variable);
+ }
+
+ & .bp { /* Builtin.Pseudo */
+
+ }
+
+ & .o,
+ & .ow { color: var(--operator); }
+
+ & .c,
+ & .cm,
+ & .cp,
+ & .c1,
+ & .cs { color: var(--comment); font-style: italic; }
+
+ /* Keywords */
+ & .k,
+ & .kc,
+ & .kd,
+ & .kn,
+ & .kp,
+ & .kr,
+ & .kt { color: var(--keyword); font-weight: 500; }
+
+ /* strings */
+ & .s,
+ & .sb,
+ & .sd,
+ & .sc,
+ & .s2,
+ & .se,
+ & .sh,
+ & .si,
+ & .sx,
+ & .sr,
+ & .ss,
+ & .s1 { color: var(--string-constant); }
+
+ /* number */
+ & .m, & .mi, & .mf { color: var(--numeric-constant); }
+
+ & .gi /* Generic.Inserted */ {
+ color: #22863a;
+ background-color: #f0fff4;
+ }
+
+ & .gd /* Generic.Deleted */ {
+ color: #b31d28;
+ background-color: #ffeef0;
+ }
+
+ & .gp /* Generic.Prompt */ {
+ color: var(--keyword);
+ font-weight: bold;
+ }
+
+ & .go /* Generic.Output */ {
+ color: #003b7e;
+ }
+
+ /* ? */
+ /*& .l,
+ & .ld,
+ & .m,
+ & .mf,
+ & .mh,
+ & .mi,
+ & .mo,
+ & .nx,
+ & .il { color: #ff0000; }*/
}
- .c { color: #93a1a1 } /* Comment */
- .err { color: #586e75 } /* Error */
- .g { color: #586e75 } /* Generic */
- .k { color: #859900 } /* Keyword */
- .l { color: #586e75 } /* Literal */
- .n { color: #586e75 } /* Name */
- .o { color: #859900 } /* Operator */
- .x { color: #cb4b16 } /* Other */
- .p { color: #586e75 } /* Punctuation */
- .cm { color: #93a1a1 } /* Comment.Multiline */
- .cp { color: #859900 } /* Comment.Preproc */
- .c1 { color: #93a1a1 } /* Comment.Single */
- .cs { color: #859900 } /* Comment.Special */
- .c, .cm, .cp, .c1, .cs {
- font-style: italic;
- }
- .gd { color: #2aa198 } /* Generic.Deleted */
- .ge { color: #586e75; font-style: italic } /* Generic.Emph */
- .gr { color: #dc322f } /* Generic.Error */
- .gh { color: #cb4b16 } /* Generic.Heading */
- .gi { color: #859900 } /* Generic.Inserted */
- .go { color: #586e75 } /* Generic.Output */
- .gp { color: #586e75 } /* Generic.Prompt */
- .gs { color: #586e75; font-weight: bold } /* Generic.Strong */
- .gu { color: #cb4b16 } /* Generic.Subheading */
- .gt { color: #586e75 } /* Generic.Traceback */
- .kc { color: #cb4b16 } /* Keyword.Constant */
- .kd { color: #268bd2 } /* Keyword.Declaration */
- .kn { color: #859900 } /* Keyword.Namespace */
- .kp { color: #859900 } /* Keyword.Pseudo */
- .kr { color: #268bd2 } /* Keyword.Reserved */
- .kt { color: #dc322f } /* Keyword.Type */
- .ld { color: #586e75 } /* Literal.Date */
- .m { color: #2aa198 } /* Literal.Number */
- .s { color: #2aa198 } /* Literal.String */
- .na { color: #586e75 } /* Name.Attribute */
- .nb { color: #B58900 } /* Name.Builtin */
- .nc { color: #268bd2 } /* Name.Class */
- .no { color: #cb4b16 } /* Name.Constant */
- .nd { color: #268bd2 } /* Name.Decorator */
- .ni { color: #cb4b16 } /* Name.Entity */
- .ne { color: #cb4b16 } /* Name.Exception */
- .nf { color: #268bd2 } /* Name.Function */
- .nl { color: #586e75 } /* Name.Label */
- .nn { color: #586e75 } /* Name.Namespace */
- .nx { color: #586e75 } /* Name.Other */
- .py { color: #586e75 } /* Name.Property */
- .nt { color: #268bd2 } /* Name.Tag */
- .nv { color: #268bd2 } /* Name.Variable */
- .ow { color: #859900 } /* Operator.Word */
- .w { color: #586e75 } /* Text.Whitespace */
- .mf { color: #2aa198 } /* Literal.Number.Float */
- .mh { color: #2aa198 } /* Literal.Number.Hex */
- .mi { color: #2aa198 } /* Literal.Number.Integer */
- .mo { color: #2aa198 } /* Literal.Number.Oct */
- .sb { color: #93a1a1 } /* Literal.String.Backtick */
- .sc { color: #2aa198 } /* Literal.String.Char */
- .sd { color: #586e75 } /* Literal.String.Doc */
- .s2 { color: #2aa198 } /* Literal.String.Double */
- .se { color: #cb4b16 } /* Literal.String.Escape */
- .sh { color: #586e75 } /* Literal.String.Heredoc */
- .si { color: #2aa198 } /* Literal.String.Interpol */
- .sx { color: #2aa198 } /* Literal.String.Other */
- .sr { color: #dc322f } /* Literal.String.Regex */
- .s1 { color: #2aa198 } /* Literal.String.Single */
- .ss { color: #2aa198 } /* Literal.String.Symbol */
- .bp { color: #268bd2 } /* Name.Builtin.Pseudo */
- .vc { color: #268bd2 } /* Name.Variable.Class */
- .vg { color: #268bd2 } /* Name.Variable.Global */
- .vi { color: #268bd2 } /* Name.Variable.Instance */
- .il { color: #2aa198 } /* Literal.Number.Integer.Long */
}
div.epigraph {
@@ -319,17 +363,6 @@
clear: both;
}
-.sans {
- font-family: $sans;
-}
-
-.exercise {
- background: rgba(38, 139, 210, 0.1);
- border-radius: 10px;
- margin: 0px -20px;
- padding: 5px 20px;
-}
-
.admonition {
.admonition-title {
font-weight: bold;
@@ -356,12 +389,14 @@
.admonition.tryit {
background: rgba(185, 246, 202, 1);
- .codehilite {
- background-color: transparent;
- color: inherit;
- margin: 0px 0px;
+ .highlight {
+ pre {
+ background-color: transparent;
+ color: inherit;
+ margin: 0px 0px;
- .c { color: grey; }
+ .c { color: grey; }
+ }
}
}
diff --git a/runtime/tools/wiki/templates/includes/favicon.html b/runtime/tools/wiki/templates/includes/favicon.html
deleted file mode 100644
index 8af739c..0000000
--- a/runtime/tools/wiki/templates/includes/favicon.html
+++ /dev/null
@@ -1,15 +0,0 @@
-<link href="https://fonts.googleapis.com/css?family=Montserrat" rel="stylesheet">
-<link rel="stylesheet" href="css/style.css" type="text/css">
-<link rel="apple-touch-icon" sizes="57x57" href="images/favicon/apple-icon-57x57.png">
-<link rel="apple-touch-icon" sizes="60x60" href="images/favicon/apple-icon-60x60.png">
-<link rel="apple-touch-icon" sizes="72x72" href="images/favicon/apple-icon-72x72.png">
-<link rel="apple-touch-icon" sizes="76x76" href="images/favicon/apple-icon-76x76.png">
-<link rel="apple-touch-icon" sizes="114x114" href="images/favicon/apple-icon-114x114.png">
-<link rel="apple-touch-icon" sizes="120x120" href="images/favicon/apple-icon-120x120.png">
-<link rel="apple-touch-icon" sizes="144x144" href="images/favicon/apple-icon-144x144.png">
-<link rel="apple-touch-icon" sizes="152x152" href="images/favicon/apple-icon-152x152.png">
-<link rel="apple-touch-icon" sizes="180x180" href="images/favicon/apple-icon-180x180.png">
-<link rel="icon" type="image/png" sizes="192x192" href="images/favicon/android-icon-192x192.png">
-<link rel="icon" type="image/png" sizes="32x32" href="images/favicon/favicon-32x32.png">
-<link rel="icon" type="image/png" sizes="96x96" href="images/favicon/favicon-96x96.png">
-<link rel="icon" type="image/png" sizes="16x16" href="images/favicon/favicon-16x16.png">
diff --git a/runtime/tools/wiki/templates/page.html b/runtime/tools/wiki/templates/page.html
index 13002c8..3b4a77f 100644
--- a/runtime/tools/wiki/templates/page.html
+++ b/runtime/tools/wiki/templates/page.html
@@ -2,9 +2,9 @@
<html>
<head>
<meta name="viewport" content="width=device-width, initial-scale=1">
+ <meta charset="utf-8">
<title>Dart VM</title>
- <link rel="stylesheet" href="/css/style.css" type="text/css">
- {% include 'includes/favicon.html' %}
+ <link rel="stylesheet" href="{{root}}/css/style.css" type="text/css">
{% include 'includes/auto-refresh.html' %}
</head>
diff --git a/runtime/tools/wiki/xref_extractor/.gitignore b/runtime/tools/wiki/xref_extractor/.gitignore
deleted file mode 100644
index 50602ac..0000000
--- a/runtime/tools/wiki/xref_extractor/.gitignore
+++ /dev/null
@@ -1,11 +0,0 @@
-# Files and directories created by pub
-.dart_tool/
-.packages
-# Remove the following pattern if you wish to check in your lock file
-pubspec.lock
-
-# Conventional directory for build outputs
-build/
-
-# Directory created by dartdoc
-doc/api/
diff --git a/runtime/tools/wiki/xref_extractor/README.md b/runtime/tools/wiki/xref_extractor/README.md
deleted file mode 100644
index 71d6af5..0000000
--- a/runtime/tools/wiki/xref_extractor/README.md
+++ /dev/null
@@ -1,54 +0,0 @@
-Tool for extracting symbolic information from C++ Dart Runtime sources using
-[cquery](https://github.com/cquery-project/cquery).
-
-It should be invoked from the root of Dart SDK checkout and will generate
-`xref.json` file containing extracted symbol information.
-
-```
-$ pushd runtime/tools/wiki/xref_extractor && pub get && popd
-$ dart runtime/tools/wiki/xref_extractor/bin/main.dart cquery/build/release/bin/cquery
-```
-
-# Prerequisites
-
-1. Build [cquery](https://github.com/cquery-project/cquery) as described [here](https://github.com/cquery-project/cquery/wiki/Building-cquery).
-2. Make sure that you have ninja files generated for ReleaseX64 configuration by
-running `tools/gn.py -a x64 -m release --no-goma` (`--no-goma` is important -
-otherwise `cquery` can't figure out which toolchain is used).
-
-# `xref.json` format
-
-```typescript
-interface Xrefs {
- /// Commit hash for which this xref.json is generated.
- commit: string;
-
- /// List of files names.
- files: string[];
-
- /// Class information by name.
- classes: ClassMap;
-
- /// Global function information.
- functions: LocationMap;
-}
-
-/// Locations are serialized as strings of form "fileIndex:lineNo", where
-/// fileIndex points into files array.
-type SymbolLocation = string;
-
-/// Information about classes is stored in an array where the first element
-/// describes location of the class itself and second optional element gives
-/// locations of class members.
-type ClassInfo = [SymbolLocation, LocationMap?];
-
-/// Map of classes by their names.
-interface ClassMap {
- [name: string]: ClassInfo;
-}
-
-/// Map of symbols to their locations.
-interface LocationMap {
- [symbol: string]: SymbolLocation;
-}
-```
\ No newline at end of file
diff --git a/runtime/tools/wiki/xref_extractor/analysis_options.yaml b/runtime/tools/wiki/xref_extractor/analysis_options.yaml
deleted file mode 100644
index c36c2c5..0000000
--- a/runtime/tools/wiki/xref_extractor/analysis_options.yaml
+++ /dev/null
@@ -1 +0,0 @@
-include: package:lints/core.yaml
diff --git a/runtime/tools/wiki/xref_extractor/bin/main.dart b/runtime/tools/wiki/xref_extractor/bin/main.dart
deleted file mode 100644
index b785dc7..0000000
--- a/runtime/tools/wiki/xref_extractor/bin/main.dart
+++ /dev/null
@@ -1,107 +0,0 @@
-// Copyright (c) 2020, the Dart project authors. Please see the AUTHORS file
-// for details. All rights reserved. Use of this source code is governed by a
-// BSD-style license that can be found in the LICENSE file.
-
-// Script used to extract symbols with locations from runtime/vm files using
-// cquery. See README.md for more information.
-
-import 'dart:convert';
-import 'dart:io';
-
-import 'package:path/path.dart' as p;
-
-import 'package:xref_extractor/cquery_driver.dart';
-import 'package:xref_extractor/xref_extractor.dart';
-
-// Note: not using Directory.createTemp to reduce indexing costs.
-const cqueryCachePath = '/tmp/cquery-cache-for-dart-sdk';
-
-void main(List<String> args) async {
- if (args.length != 1 || !File(args[0]).existsSync()) {
- print('''
-Usage: dart runtime/tools/wiki/xref_extractor/bin/main.dart <path-to-cquery>
-''');
- exit(1);
- }
-
- final cqueryBinary = args[0];
-
- // Sanity check that we are running from SDK checkout root.
- final sdkCheckoutRoot = Directory.current.absolute;
- final runtimeVmDirectory =
- Directory(p.join(sdkCheckoutRoot.path, 'runtime/vm'));
- final gitDirectory = Directory(p.join(sdkCheckoutRoot.path, '.git'));
-
- if (!gitDirectory.existsSync() || !runtimeVmDirectory.existsSync()) {
- print('This script expects to be run from SDK checkout root');
- exit(1);
- }
-
- // Generate compile_commands.json from which cquery will extract compilation
- // flags for individual C++ files.
- await generateCompileCommands();
-
- // Start cquery process and request indexing of runtimeVmDirectory.
- final cquery = await CqueryDriver.start(cqueryBinary);
-
- print('Indexing ${runtimeVmDirectory.path} with cquery');
- cquery.progress.listen((files) =>
- stdout.write('\rcquery is running ($files files left to index)'));
-
- await cquery.request('initialize', params: {
- 'processId': 123,
- 'rootUri': sdkCheckoutRoot.uri.toString(),
- 'capabilities': {
- 'textDocument': {'codeLens': null}
- },
- 'trace': 'on',
- 'initializationOptions': {
- 'cacheDirectory': cqueryCachePath,
- 'progressReportFrequencyMs': 1000,
- },
- 'workspaceFolders': [
- {
- 'uri': runtimeVmDirectory.uri.toString(),
- 'name': 'vm',
- }
- ]
- });
-
- // Tell cquery to wait for the indexing to complete and then exit.
- cquery.notify(r'$cquery/wait');
- cquery.notify(r'exit');
-
- // Wait for cquery to exit.
- final exitCode = await cquery.exitCode;
- print('\r\x1b[K... completed (cquery exited with exit code ${exitCode})');
-
- // Process cquery cache folder to extract symbolic information.
- await generateXRef(cqueryCachePath, sdkCheckoutRoot.path,
- (path) => path.startsWith('runtime/'));
-}
-
-/// Generate compile_commands.json for cquery so that it could index VM sources.
-///
-/// We ask ninja to produce compilation database for X64 release build and then
-/// post process it to limit it to libdart_vm_precompiler_host_targeting_host
-/// target, because otherwise we get duplicated compilation commands for the
-/// same input C++ files and this greatly confuses cquery.
-Future<void> generateCompileCommands() async {
- print('Extracting compilation commands from build files for ReleaseX64');
- final result = await Process.run('buildtools/ninja/ninja', [
- '-C',
- '${Platform.isMacOS ? 'xcodebuild' : 'out'}/ReleaseX64',
- '-t',
- 'compdb',
- 'cxx'
- ]);
- final List<dynamic> commands = jsonDecode(result.stdout);
- final re = RegExp(r'/libdart(_vm)?_precompiler_host_targeting_host\.');
- final filteredCommands = commands
- .cast<Map<String, dynamic>>()
- .where((item) => item['command'].contains(re))
- .toList(growable: false);
- File('compile_commands.json').writeAsStringSync(jsonEncode(filteredCommands));
- print('''
-... generated compile_commands.json with ${filteredCommands.length} entries''');
-}
diff --git a/runtime/tools/wiki/xref_extractor/lib/cquery_driver.dart b/runtime/tools/wiki/xref_extractor/lib/cquery_driver.dart
deleted file mode 100644
index 021a4ff..0000000
--- a/runtime/tools/wiki/xref_extractor/lib/cquery_driver.dart
+++ /dev/null
@@ -1,200 +0,0 @@
-// Copyright (c) 2020, the Dart project authors. Please see the AUTHORS file
-// for details. All rights reserved. Use of this source code is governed by a
-// BSD-style license that can be found in the LICENSE file.
-
-// Library for spawning cquery process and communicating with it.
-
-import 'dart:async';
-import 'dart:convert';
-import 'dart:io';
-import 'dart:math' as math;
-
-/// Helper class encapsulating communication with the cquery process.
-///
-/// cquery communicates using jsonrpc via stdin/stdout streams.
-class CqueryDriver {
- final Process _cquery;
-
- final _pendingRequests = <int, Completer<dynamic>>{};
- var _requestId = 1;
-
- final _progressStreamController = StreamController<int>.broadcast();
-
- CqueryDriver._(this._cquery) {
- _cquery.stdout
- .transform(utf8.decoder)
- .transform(_JsonRpcParser.transformer)
- .listen(_handleMessage);
- _cquery.stderr
- .transform(utf8.decoder)
- .listen((data) => print('stderr: ${data}'));
- }
-
- static Future<CqueryDriver> start(String cqueryBinary) async {
- final cquery = await Process.start(cqueryBinary, ['--language-server']);
- return CqueryDriver._(cquery);
- }
-
- /// Stream of progress notifications from cquery. Indicate how many files
- /// are currently pending indexing.
- Stream<int> get progress => _progressStreamController.stream;
-
- Future<int> get exitCode => _cquery.exitCode;
-
- /// Send the given message to the cquery process.
- void _sendMessage(Map<String, dynamic> data) {
- data['jsonrpc'] = '2.0';
-
- final rq = jsonEncode(data);
- _cquery.stdin.write('Content-Length: ${rq.length}\r\n\r\n${rq}');
- }
-
- /// Send a notification message to the cquery process.
- void notify(String method, {Map<String, dynamic> params}) {
- final rq = <String, dynamic>{'method': method};
- if (params != null) rq['params'] = params;
- _sendMessage(rq);
- }
-
- /// Send a request message to the cquery process.
- Future<dynamic> request(String method, {Map<String, dynamic> params}) {
- final rq = <String, dynamic>{'method': method, 'id': _requestId++};
- _pendingRequests[rq['id']] = Completer();
- if (params != null) rq['params'] = params;
- _sendMessage(rq);
- return _pendingRequests[rq['id']].future;
- }
-
- /// Handle message received from cquery process.
- void _handleMessage(Map<String, dynamic> data) {
- final method = data['method'];
-
- // If it is a progress notification issue progress event.
- if (method == r'$cquery/progress') {
- _progressStreamController.add(data['params']['indexRequestCount']);
- }
-
- // Otherwise check if it is a response to one of our requests and complete
- // corresponding future if it is.
- if (data.containsKey('id') && data.containsKey('result')) {
- final id = data['id'];
- final result = data['result'];
- _pendingRequests[id].complete(result);
- _pendingRequests[id] = null;
- return;
- }
- }
-}
-
-/// Simple parser for jsonrpc protocol over arbitrary chunked stream.
-class _JsonRpcParser {
- /// Accumulator for the message content.
- final StringBuffer content = StringBuffer();
-
- /// Number of bytes left to read in the current message.
- int pendingContentLength = 0;
-
- /// Auxiliary variable to store various state between invocations of
- /// [state] callback.
- int index = 0;
-
- /// Position inside incoming chunk of data.
- int pos = 0;
-
- /// Current parser state.
- Function state = _JsonRpcParser.readHeader;
-
- /// Callback to invoke when we finish parsing complete message.
- void Function(Map<String, dynamic>) onMessage;
-
- _JsonRpcParser({this.onMessage});
-
- /// StreamTransformer wrapping _JsonRpcParser.
- static StreamTransformer<String, Map<String, dynamic>> get transformer =>
- StreamTransformer.fromBind((Stream<String> s) {
- final output = StreamController<Map<String, dynamic>>();
- final p = _JsonRpcParser(onMessage: output.add);
- s.listen(p.addChunk);
- return output.stream;
- });
-
- /// Parse the chunk of data.
- void addChunk(String data) {
- pos = 0;
- while (pos < data.length) {
- state = state(this, data);
- }
- }
-
- /// Parsing state: waiting for 'Content-Length' header.
- static Function readHeader(_JsonRpcParser p, String data) {
- final codeUnit = data.codeUnitAt(p.pos++);
-
- if (HEADER.codeUnitAt(p.index) != codeUnit) {
- throw 'Unexpected codeUnit: ${String.fromCharCode(codeUnit)} expected ${HEADER[p.index]}';
- }
-
- p.index++;
- if (p.index == HEADER.length) {
- p.index = 0;
- return _JsonRpcParser.readLength;
- }
- return _JsonRpcParser.readHeader;
- }
-
- /// Parsing state: parsing content length value.
- static Function readLength(_JsonRpcParser p, String data) {
- final codeUnit = data.codeUnitAt(p.pos++);
-
- if (codeUnit == CR) {
- p.pendingContentLength = p.index;
- p.index = 0;
- return _JsonRpcParser.readHeaderEnd;
- }
- if (codeUnit < CH0 || codeUnit > CH9) {
- throw 'Unexpected codeUnit: ${String.fromCharCode(codeUnit)} expected 0 to 9';
- }
- p.index = p.index * 10 + (codeUnit - CH0);
- return _JsonRpcParser.readLength;
- }
-
- /// Parsing state: content length was read, skipping line breaks before
- /// content start.
- static Function readHeaderEnd(_JsonRpcParser p, String data) {
- final codeUnit = data.codeUnitAt(p.pos++);
-
- if (HEADER_END.codeUnitAt(p.index) != codeUnit) {
- throw 'Unexpected codeUnit: ${String.fromCharCode(codeUnit)} expected ${HEADER_END[p.index]}';
- }
-
- p.index++;
- if (p.index == HEADER_END.length) {
- return _JsonRpcParser.readContent;
- }
- return _JsonRpcParser.readHeaderEnd;
- }
-
- /// Parsing state: reading message content.
- static Function readContent(_JsonRpcParser p, String data) {
- final availableBytes = data.length - p.pos;
- final bytesToRead = math.min(availableBytes, p.pendingContentLength);
- p.content.write(data.substring(p.pos, p.pos + bytesToRead));
- p.pendingContentLength -= bytesToRead;
- p.pos += bytesToRead;
- if (p.pendingContentLength == 0) {
- p.onMessage(jsonDecode(p.content.toString()));
- p.content.clear();
- p.index = 0;
- return _JsonRpcParser.readHeader;
- } else {
- return _JsonRpcParser.readContent;
- }
- }
-
- static const HEADER = 'Content-Length: ';
- static const HEADER_END = '\n\r\n';
- static const CH0 = 48;
- static const CH9 = 57;
- static const CR = 13;
- static const LF = 10; // ignore: unused_field
-}
diff --git a/runtime/tools/wiki/xref_extractor/lib/xref_extractor.dart b/runtime/tools/wiki/xref_extractor/lib/xref_extractor.dart
deleted file mode 100644
index 6f0b033..0000000
--- a/runtime/tools/wiki/xref_extractor/lib/xref_extractor.dart
+++ /dev/null
@@ -1,235 +0,0 @@
-// Copyright (c) 2020, the Dart project authors. Please see the AUTHORS file
-// for details. All rights reserved. Use of this source code is governed by a
-// BSD-style license that can be found in the LICENSE file.
-
-// Library for converting cquery cache into condensed symbol location
-// information expected by our xref markdown extension.
-
-import 'dart:convert';
-import 'dart:io';
-import 'dart:async';
-
-import 'package:path/path.dart' as p;
-
-typedef FileFilterCallback = bool Function(String path);
-
-/// Load cquery cache from the given directory and accumulate all symbols
-/// (global functions and class methods) for the Dart SDK related sources
-/// matching the given filter.
-Future<void> generateXRef(String cqueryCachePath, String sdkRootPath,
- FileFilterCallback fileFilter) async {
- final cacheRoot =
- Directory(p.join(cqueryCachePath, sdkRootPath.replaceAll('/', '@')));
- final files = cacheRoot
- .listSync()
- .whereType<File>()
- .where((file) => file.path.endsWith('.json'))
- .toList();
- print('Processing ${files.length} indexes available in ${cacheRoot.path}');
-
- final cache = CqueryCache(fileFilter: fileFilter);
- files.forEach(cache.loadFile);
-
- final classesByName = <String, Class>{
- for (var entry in cache.classes.entries)
- if (entry.value.name != null && entry.value.name != '')
- entry.value.name: entry.value
- };
-
- final database = {
- 'commit': await currentCommitHash(),
- 'files': cache.filesByIndex,
- 'classes': classesByName,
- 'functions': cache.globals.uniqueMembers
- };
-
- File('xref.json').writeAsStringSync(jsonEncode(database));
- print('... done (written xref.json)');
-}
-
-/// Helper class representing symbol information contained in the cquery cache.
-class CqueryCache {
- final files = <String, int>{};
- final filesByIndex = [];
-
- final classes = <num, Class>{};
- final globals = Class('\$Globals');
-
- FileFilterCallback fileFilter;
-
- CqueryCache({this.fileFilter});
-
- int addFile(String name) => files.putIfAbsent(name, () {
- filesByIndex.add(name);
- return filesByIndex.length - 1;
- });
-
- Location makeLocation(String file, int lineNo) =>
- Location(addFile(file), lineNo);
-
- // cquery used to serialize USRs as integers but they are now serialized as
- // doubles (with .0 at the end) for some reason. This might even lead to
- // incorrect deserialization with a loss of a precise USR value - but
- // should not lead to any issues as long as two different classes don't
- // have conflicting USRs.
- Class findClassByUsr(num usr) => classes.putIfAbsent(usr, () => Class());
-
- void defineClass(num usr, String name, Location loc) {
- final cls = findClassByUsr(usr);
- if (cls.name != null && cls.name != '' && cls.name != name) {
- throw 'Mismatched names';
- }
- if (name != '') cls.name = name;
- if (cls.loc == null) {
- cls.loc = loc;
- } else {
- cls.loc = Location.invalid;
- }
- }
-
- void loadFile(File indexFile) {
- final result = jsonDecode(indexFile.readAsStringSync().split('\n')[1]);
-
- // Check if we are interested in the original source file.
- final sourceFile =
- p.basenameWithoutExtension(indexFile.path).replaceAll('@', '/');
- if (!fileFilter(sourceFile)) return;
-
- // Extract classes defined in the file.
- for (var type in result['types']) {
- if (type['kind'] != SymbolKind.Class) continue;
- final extent = type['extent'];
- if (extent == null) continue;
-
- final detailedName = type['detailed_name'];
- final lineStart = int.parse(extent.substring(0, extent.indexOf(':')));
- defineClass(
- type['usr'], detailedName, makeLocation(sourceFile, lineStart));
- }
-
- // Extract class methods defined in the file.
- for (var func in result['funcs']) {
- final kind = func['kind'];
- if (kind != SymbolKind.Method && kind != SymbolKind.StaticMethod) {
- continue;
- }
- final extent = func['extent'];
- if (extent == null) continue;
- final short = shortName(func);
- final lineStart = int.parse(extent.substring(0, extent.indexOf(':')));
- if (func['declaring_type'] == null) {
- continue;
- }
- findClassByUsr(result['types'][func['declaring_type']]['usr'])
- .defineMember(short, makeLocation(sourceFile, lineStart));
- }
-
- // Extract global functions defined in the file.
- for (var func in result['funcs']) {
- final kind = func['kind'];
- if (kind != SymbolKind.Function) continue;
- final extent = func['extent'];
- if (extent == null) continue;
- final short = shortName(func);
- final lineStart = int.parse(extent.substring(0, extent.indexOf(':')));
- globals.defineMember(short, makeLocation(sourceFile, lineStart));
- }
- }
-}
-
-class Class {
- String name;
- Location loc;
-
- // Member to definition location map. If the same symbol has multiple
- // definitions then we mark it with [Location.invalid].
- Map<String, Location> members;
-
- Class([this.name]);
-
- void defineMember(String name, Location loc) {
- members ??= <String, Location>{};
- members[name] = members.containsKey(name) ? Location.invalid : loc;
- }
-
- dynamic toJson() {
- final result = [loc?.toJson() ?? 0];
- if (members != null) {
- final res = uniqueMembers;
- if (res.isNotEmpty) {
- result.add(res);
- }
- }
- return result;
- }
-
- Map<String, Location> get uniqueMembers => <String, Location>{
- for (var entry in members.entries)
- if (entry.value != Location.invalid) entry.key: entry.value
- };
-}
-
-class Location {
- final int file;
- final int lineNo;
-
- const Location(this.file, this.lineNo);
-
- String toJson() => identical(this, invalid) ? null : '$file:$lineNo';
-
- @override
- String toString() => '$file:$lineNo';
-
- static const invalid = Location(-1, -1);
-}
-
-String shortName(entity) {
- final offset = entity['short_name_offset'];
- final length = entity['short_name_size'] ?? 0;
- final detailedName = entity['detailed_name'];
- if (length == 0) return detailedName;
- return detailedName.substring(offset, offset + length);
-}
-
-Future<String> currentCommitHash() async {
- final results = await Process.run('git', ['rev-parse', 'HEAD']);
- return results.stdout;
-}
-
-/// Kind of the symbol. Taken from LSP specifications and cquery source code.
-abstract class SymbolKind {
- static const Unknown = 0;
-
- static const File = 1;
- static const Module = 2;
- static const Namespace = 3;
- static const Package = 4;
- static const Class = 5;
- static const Method = 6;
- static const Property = 7;
- static const Field = 8;
- static const Constructor = 9;
- static const Enum = 10;
- static const Interface = 11;
- static const Function = 12;
- static const Variable = 13;
- static const Constant = 14;
- static const String = 15;
- static const Number = 16;
- static const Boolean = 17;
- static const Array = 18;
- static const Object = 19;
- static const Key = 20;
- static const Null = 21;
- static const EnumMember = 22;
- static const Struct = 23;
- static const Event = 24;
- static const Operator = 25;
- static const TypeParameter = 26;
-
- // cquery extensions.
- static const TypeAlias = 252;
- static const Parameter = 253;
- static const StaticMethod = 254;
- static const Macro = 255;
-}
diff --git a/runtime/tools/wiki/xref_extractor/pubspec.yaml b/runtime/tools/wiki/xref_extractor/pubspec.yaml
deleted file mode 100644
index 11fd5ef..0000000
--- a/runtime/tools/wiki/xref_extractor/pubspec.yaml
+++ /dev/null
@@ -1,13 +0,0 @@
-name: xref_extractor
-description: A sample command-line application.
-# This package is not intended for consumption on pub.dev. DO NOT publish.
-publish_to: none
-
-environment:
- sdk: '>=2.7.0 <3.0.0'
-
-dependencies:
- path: any
-
-dev_dependencies:
- lints: any
To view, visit change 280262. To unsubscribe, or for help writing mail filters, visit settings.
Slava Egorov (Gerrit)
Feb 15, 2023, 8:46:30 AM2/15/23
to rev...@dartlang.org, vm-...@dartlang.org, Martin Kustermann
Attention is currently required from: Martin Kustermann.
1 comment:
Patchset:
This is something I have done around Dart Summit in October - but never got to landing. Want to land it now so that it does not rot away.
To view, visit change 280262. To unsubscribe, or for help writing mail filters, visit settings.
Martin Kustermann (Gerrit)
Feb 16, 2023, 7:48:58 AM2/16/23
to Slava Egorov, rev...@dartlang.org, vm-...@dartlang.org
Attention is currently required from: Slava Egorov.
Patch set 4:Code-Review +1
1 comment:
File runtime/docs/index.md:
Patch Set #4, Line 71:
httpsMaybe avoid baking in version numbers, instead use e.g. `https://api.dartlang.org/stable/dart-isolate/Isolate/spawnUri.html`
To view, visit change 280262. To unsubscribe, or for help writing mail filters, visit settings.
Slava Egorov (Gerrit)
Feb 16, 2023, 7:52:12 AM2/16/23
to rev...@dartlang.org, vm-...@dartlang.org, Martin Kustermann
1 comment:
File runtime/docs/index.md:
Patch Set #4, Line 71:
https
Maybe avoid baking in version numbers, instead use e.g. `https://api.dartlang. […]
Done
To view, visit change 280262. To unsubscribe, or for help writing mail filters, visit settings.
Slava Egorov (Gerrit)
Feb 16, 2023, 7:52:40 AM2/16/23
to rev...@dartlang.org, vm-...@dartlang.org, Martin Kustermann
Patch set 5:Commit-Queue +2
Commit Queue (Gerrit)
Feb 16, 2023, 8:42:08 AM2/16/23
to Slava Egorov, rev...@dartlang.org, vm-...@dartlang.org, Martin Kustermann
Commit Queue submitted this change.
4 is the latest approved patch-set.
The change was submitted with unreviewed changes in the following files:
```
The name of the file: runtime/docs/index.md
Insertions: 1, Deletions: 1.
@@ -68,7 +68,7 @@
Any Dart code within the VM is running within some _isolate_, which can be best described as an isolated Dart universe with its own global state and _usually_ with its own thread of control (*mutator thread*). Isolates are grouped together into _isolate groups_. Isolate within the group share the same garbage collector managed *heap*, used as a storage for objects allocated by an isolate. Heap sharing between isolates in the same group is an implementation detail which is not observable from the Dart code. Even isolates within the same group can not share any mutable state directly and can only communicate by message passing through *ports* (not to be confused with network ports!).
-Isolates within a group share the same Dart program. [`Isolate.spawn`](https://api.dart.dev/stable/dart-isolate/Isolate/spawn.html) spawns an isolate within the same group, while [`Isolate.spawnUri`](https://api.dart.dev/stable/2.18.2/dart-isolate/Isolate/spawnUri.html) starts a new group.
+Isolates within a group share the same Dart program. [`Isolate.spawn`](https://api.dart.dev/stable/dart-isolate/Isolate/spawn.html) spawns an isolate within the same group, while [`Isolate.spawnUri`](https://api.dart.dev/stable/dart-isolate/Isolate/spawnUri.html) starts a new group.
The relationship between OS threads and isolates is a bit blurry and highly dependent on how VM is embedded into an application. Only the following is guaranteed:
```
Approvals:
Martin Kustermann: Looks good to me, approved
Slava Egorov: Commit
[vm/docs] Improve VM docs tooling.
Main change is around removing our custom syntax which allows
to read markdown files directly on GitHub:
* instead of using custom link format `@{xref}` we start
using normal links. For example, [`dart::ThreadPool`][] is
understood as a ref to `dart::ThreadPool` class declaration.
`build.py` script injects an xref section at the end of
each markdown file.
* similarly we don't use custom syntax for admonitions, but
instead use blockquotes. `build.py` detects block quotes
which start with a marker like `**Note**` and renders
then in a custom way.
This CL also drops dependency on cquery and instead rewrites
indexing in pure Python via libclang.
Change-Id: I0b47ec93f632de89627a3c682d511c8b86c58430
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/280262
Reviewed-by: Martin Kustermann <kuste...@google.com>
Commit-Queue: Slava Egorov <veg...@google.com>
39 files changed, 1,526 insertions(+), 1,176 deletions(-)
index 21a9843..9d76b10 100644+Isolates within a group share the same Dart program. [`Isolate.spawn`](https://api.dart.dev/stable/dart-isolate/Isolate/spawn.html) spawns an isolate within the same group, while [`Isolate.spawnUri`](https://api.dart.dev/stable/dart-isolate/Isolate/spawnUri.html) starts a new group.To view, visit change 280262. To unsubscribe, or for help writing mail filters, visit settings.
CBuild (Gerrit)
Feb 16, 2023, 9:20:36 AM2/16/23
to Commit Queue, Slava Egorov, rev...@dartlang.org, vm-...@dartlang.org, Martin Kustermann
go/dart-cbuild result: SUCCESS
Details: https://goto.google.com/dart-cbuild/find/b2d5245dcd94707d4edeb60e501f995fa3b74b36
Reply all
Reply to author
Forward
0 new messages