Dart Analyzer Plugin

[Norbert Kozsir]

Hi,

I started working on a dart analyzer plugin for Flutter which helps deal with the bloc library.

Unfortunately the analyzer API seems pretty work on progress and I wasn't able to find too much information about it online. I got a version working, but it is very inefficient.
Basically, what I need to do is analyze the whole project to find relevant subclasses of "Bloc" and a few other things which need access to the whole project.

I found "getResolvedLibrary" to resolve the whole library but it didn't seem to work. Therefore I resorted to analyzing every file in the project - every time a request comes in.

for (var path in request.result.session.analysisContext.contextRoot.analyzedFiles().where((it) => it.endsWith(".dart"))) {
  var unitElement = await request.result.session.getResolvedUnit(path);
  if (!unitElement.isPart) {
    logger.log('Analyzing $path ....');
    unitElement.libraryElement.accept(blocStateCollector);
  }
}

(https://github.com/Norbert515/bloc_builder_host_package/blob/master/tools/analyzer_plugin/bin/event_navigation.dart#L64-L71 - sorry about the messy code)

This totally worked for small projects with few files but failed on large ones.

I was hoping that "getResolvedUnit" did some internal caching - but I'm very unsure about that.

My question is, how does one go about analyzing the whole project? Are there any resources besides https://github.com/dart-lang/sdk/blob/master/pkg/analyzer_plugin/doc/tutorial/tutorial.md that could help with this?

I feel like there is a lot of power in analyzer plugins and I'd love to contribute in some way!


Thanks,
Norbert

[Brian Wilkerson]

Unfortunately the analyzer API seems pretty work on progress and I wasn't able to find too much information about it online.

I'm not sure which API you're referring to. It's true that the API in the analyzer package has been transitioning toward using `AnalysisContextCollection` and the code reachable from it, and that it can be hard to know which APIs are now, or soon will be, obsolete. In part that's because the Dart language continues to evolve, so the requirements for analyzing it continue to change over time. We have a long-standing goal of cleaning up the API, but maintaining correct analysis in the face of a changing language is higher priority.

Basically, what I need to do is analyze the whole project to find relevant subclasses of "Bloc" and a few other things which need access to the whole project.

That's kind of vague, so my response will necessarily be fairly generic. If you'd like to share more information (such as what information you need and what you're using it for) I might be able to provide more specific advice.

I found "getResolvedLibrary" to resolve the whole library but it didn't seem to work.

Can you provide more details about what specifically didn't work?

Therefore I resorted to analyzing every file in the project - every time a request comes in.

This totally worked for small projects with few files but failed on large ones.

I was hoping that "getResolvedUnit" did some internal caching - but I'm very unsure about that.

No, `getResolvedUnit` doesn't do any caching.

My question is, how does one go about analyzing the whole project? Are there any resources besides https://github.com/dart-lang/sdk/blob/master/pkg/analyzer_plugin/doc/tutorial/tutorial.md that could help with this?

I'm not aware of any written answer for this question, but it does suggest that we should add something to the tutorial. It's an important topic for plugin developers.

The short answer is that you don't analyze the whole project. At least not on every request. The analysis server is a long-lived process, which means that the plugins it runs are also long-lived. The way plugins are expected to work is that they should analyze all of the files in the context roots once, when the `analysis.setContextRoots` request is received, sending the resulting information back to the server, and then to incrementally update the information sent back to the server as individual files are modified (when an `analysis.handleWatchEvents` or `analysis.updateContent` request is received).

If your plugin needs information from outside the immediate library being analyzed, then you'll need to cache that information when you perform the initial analysis after the context roots are set. You'll also need to track the dependencies between files so that if a file that contributed to the cached information is changed you can update the cache and then re-analyze all of the libraries whose results depended on the content of the cache. Unfortunately, building a cache for information like this is hard to get right, and cache management can also lead to performance issues. For example, you can't store either AST nodes, elements from the element model, or types in the cache because all of those will lead you to hold on to too much memory.

I feel like there is a lot of power in analyzer plugins and I'd love to contribute in some way!

Contributions are always welcome!

--
You received this message because you are subscribed to the Google Groups "Dart Analyzer Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to analyzer-discu...@dartlang.org.
To view this discussion on the web visit https://groups.google.com/a/dartlang.org/d/msgid/analyzer-discuss/ce1dc767-a281-4ab1-b332-cb65fbde78b7%40dartlang.org.

[Norbert Kozsir]

Thanks for the quick response!

 

That's kind of vague, so my response will necessarily be fairly generic. If you'd like to share more information (such as what information you need and what you're using it for) I might be able to provide more specific advice.

I got the analysis of the tree working, my main difficulty was accessing that tree and working with the analyzer plugin API.

Right now I have two features I'd like the plugin to support. 

1. Wrap with BlocBulder

Similar to "Wrap with StreamBuilder", but with the addition that the quick action helps with the generic BlocBuilder<Bloc, BlocState> arguments by scanning the project and providing 

linkEditGroups with suggestions.

2. Navigating to event processing

With the bloc package, you usually dispatch events by calling "bloc.add(MyEvent())". When invoking the "go to definition" on the event, the IDE takes you to the class element of that event.

Most of the time you are interested in the method where this event is processed though. This plugin will, in addition, suggest the navigation to the mapEventToState method in its respective bloc class.

class MyBloc .... {

  @override
  Stream<States> mapEventToState(Event event) {
    if(event is MyEvent) {}
  }


}

Can you provide more details about what specifically didn't work?

 

class MyVisitor extends RecursiveElementVisitor {

  @override

  visitClassElement(ClassElement element) {

    print("HI!");

    return super.visitClassElement(element);

  }

}

void main() async {

  List<String> includedPaths = <String>["C:\\Users\\Norbert\\workspace\\analysis_plugin\\bloc_builder_test_project\\lib"];

  AnalysisContextCollection collection = new AnalysisContextCollection(

      includedPaths: includedPaths,

    resourceProvider: PhysicalResourceProvider.INSTANCE

  );

  var library = await collection.

    contextFor("C:\\Users\\Norbert\\workspace\\analysis_plugin\\bloc_builder_test_project\\lib").

    currentSession.getResolvedLibrary("C:\\Users\\Norbert\\workspace\\analysis_plugin\\bloc_builder_test_project\\lib");

  library.element.visitChildren(MyVisitor());

}

That lib folder contains multiple files with multiple classes.

Reading the documentation I would expect the visitor to visit all class declarations.

  /// Return a future that will complete with information about the results of

  /// resolving all of the files in the library with the given absolute,

  /// normalized [path].

  ///

  /// Throw [ArgumentError] if the given [path] is not the defining compilation

  /// unit for a library (that is, is a part of a library).

Unfortunately, that doesn't seem to be the case.

On the other hand, when looping through: 

session.analysisContext.contextRoot.analyzedFiles()

and calling

var unitElement = await request.result.session.getResolvedUnit(path);

on each file, it contains all elements.

I'm not aware of any written answer for this question, but it does suggest that we should add something to the tutorial. It's an important topic for plugin developers.

I'd be happy to contribute! 

The short answer is that you don't analyze the whole project. At least not on every request. The analysis server is a long-lived process, which means that the plugins it runs are also long-lived. The way plugins are expected to work is that they should analyze all of the files in the context roots once, when the `analysis.setContextRoots` request is received, sending the resulting information back to the server, and then to incrementally update the information sent back to the server as individual files are modified (when an `analysis.handleWatchEvents` or `analysis.updateContent` request is received).

If your plugin needs information from outside the immediate library being analyzed, then you'll need to cache that information when you perform the initial analysis after the context roots are set. You'll also need to track the dependencies between files so that if a file that contributed to the cached information is changed you can update the cache and then re-analyze all of the libraries whose results depended on the content of the cache. Unfortunately, building a cache for information like this is hard to get right, and cache management can also lead to performance issues. For example, you can't store either AST nodes, elements from the element model, or types in the cache because all of those will lead you to hold on to too much memory.

The actual information I need to cache is pretty trivial (names and locations of class declarations).

So what I've got right now:

  @override

  AnalysisDriverGeneric createAnalysisDriver(plugin.ContextRoot contextRoot) {

    var root = ContextRoot(contextRoot.root, contextRoot.exclude,

        pathContext: resourceProvider.pathContext)

      ..optionsFilePath = contextRoot.optionsFile;

    var contextBuilder = ContextBuilder(resourceProvider, sdkManager, null)

      ..analysisDriverScheduler = analysisDriverScheduler

      ..byteStore = byteStore

      ..performanceLog = performanceLog

      ..fileContentOverlay = fileContentOverlay;

    var result = contextBuilder.buildDriver(root);

    result.results.listen(_processResult);

    return result;

  }

  @override

  void contentChanged(String path) {

    super.driverForPath(path).addFile(path);

  }

I've copied this code from other plugins, therefore I'm not 100% sure what exactly each component of this does.

The "ServerPlugin" seems to handle the setContextRoots.

  /**

   * Handle an 'analysis.setContextRoots' request.

   *

   * Throw a [RequestFailure] if the request could not be handled.

   */

  Future<AnalysisSetContextRootsResult> handleAnalysisSetContextRoots(

      AnalysisSetContextRootsParams parameters) async {

    // TODO(brianwilkerson) Determine whether this await is necessary.

    await null;

    List<ContextRoot> contextRoots = parameters.roots;

    List<ContextRoot> oldRoots = driverMap.keys.toList();

    for (ContextRoot contextRoot in contextRoots) {

      if (!oldRoots.remove(contextRoot)) {

        // The context is new, so we create a driver for it. Creating the driver

        // has the side-effect of adding it to the analysis driver scheduler.

        AnalysisDriverGeneric driver = createAnalysisDriver(contextRoot);

        driverMap[contextRoot] = driver;

        _addFilesToDriver(

            driver,

            resourceProvider.getResource(contextRoot.root),

            contextRoot.exclude);

      }

    }

    for (ContextRoot contextRoot in oldRoots) {

      // The context has been removed, so we remove its driver.

      AnalysisDriverGeneric driver = driverMap.remove(contextRoot);

      // The `dispose` method has the side-effect of removing the driver from

      // the analysis driver scheduler.

      driver.dispose();

    }

    return new AnalysisSetContextRootsResult();

  }

Following your suggestion, I found the "updateContent" method

  /**

   * Handle an 'analysis.updateContent' request. Most subclasses should not

   * override this method, but should instead use the [contentCache] to access

   * the current content of overlaid files.

   *

   * Throw a [RequestFailure] if the request could not be handled.

   */

  Future<AnalysisUpdateContentResult> handleAnalysisUpdateContent(

      AnalysisUpdateContentParams parameters) async {

    // TODO(brianwilkerson) Determine whether this await is necessary.

    await null;

    Map<String, Object> files = parameters.files;

    files.forEach((String filePath, Object overlay) {

      if (overlay is AddContentOverlay) {

        fileContentOverlay[filePath] = overlay.content;

      } else if (overlay is ChangeContentOverlay) {

        String oldContents = fileContentOverlay[filePath];

        String newContents;

        if (oldContents == null) {

          // The server should only send a ChangeContentOverlay if there is

          // already an existing overlay for the source.

          throw new RequestFailure(

              RequestErrorFactory.invalidOverlayChangeNoContent());

        }

        try {

          newContents = SourceEdit.applySequence(oldContents, overlay.edits);

        } on RangeError {

          throw new RequestFailure(

              RequestErrorFactory.invalidOverlayChangeInvalidEdit());

        }

        fileContentOverlay[filePath] = newContents;

      } else if (overlay is RemoveContentOverlay) {

        fileContentOverlay[filePath] = null;

      }

      contentChanged(filePath);

    });

    return new AnalysisUpdateContentResult();

  }

So from what I understood, I should perform the initial whole project analysis in "createAnalysisDriver" and then re-analyze the files in "contentChanged"?

As a simple example, how would I go about creating a plugin which caches all the names of classes in the project?

Thanks again! Looking forward to contributing. 

 

Am Donnerstag, 21. Mai 2020 17:24:33 UTC+2 schrieb brianwilkerson:

Unfortunately the analyzer API seems pretty work on progress and I wasn't able to find too much information about it online.

I'm not sure which API you're referring to. It's true that the API in the analyzer package has been transitioning toward using `AnalysisContextCollection` and the code reachable from it, and that it can be hard to know which APIs are now, or soon will be, obsolete. In part that's because the Dart language continues to evolve, so the requirements for analyzing it continue to change over time. We have a long-standing goal of cleaning up the API, but maintaining correct analysis in the face of a changing language is higher priority.

Basically, what I need to do is analyze the whole project to find relevant subclasses of "Bloc" and a few other things which need access to the whole project.

That's kind of vague, so my response will necessarily be fairly generic. If you'd like to share more information (such as what information you need and what you're using it for) I might be able to provide more specific advice.

I found "getResolvedLibrary" to resolve the whole library but it didn't seem to work.

Can you provide more details about what specifically didn't work?

Therefore I resorted to analyzing every file in the project - every time a request comes in.

This totally worked for small projects with few files but failed on large ones.

I was hoping that "getResolvedUnit" did some internal caching - but I'm very unsure about that.

No, `getResolvedUnit` doesn't do any caching.

My question is, how does one go about analyzing the whole project? Are there any resources besides https://github.com/dart-lang/sdk/blob/master/pkg/analyzer_plugin/doc/tutorial/tutorial.md that could help with this?

I'm not aware of any written answer for this question, but it does suggest that we should add something to the tutorial. It's an important topic for plugin developers.

The short answer is that you don't analyze the whole project. At least not on every request. The analysis server is a long-lived process, which means that the plugins it runs are also long-lived. The way plugins are expected to work is that they should analyze all of the files in the context roots once, when the `analysis.setContextRoots` request is received, sending the resulting information back to the server, and then to incrementally update the information sent back to the server as individual files are modified (when an `analysis.handleWatchEvents` or `analysis.updateContent` request is received).

If your plugin needs information from outside the immediate library being analyzed, then you'll need to cache that information when you perform the initial analysis after the context roots are set. You'll also need to track the dependencies between files so that if a file that contributed to the cached information is changed you can update the cache and then re-analyze all of the libraries whose results depended on the content of the cache. Unfortunately, building a cache for information like this is hard to get right, and cache management can also lead to performance issues. For example, you can't store either AST nodes, elements from the element model, or types in the cache because all of those will lead you to hold on to too much memory.

I feel like there is a lot of power in analyzer plugins and I'd love to contribute in some way!

Contributions are always welcome!

On Thu, May 21, 2020 at 7:04 AM Norbert Kozsir <kozsir...@gmail.com> wrote:

Hi,

I started working on a dart analyzer plugin for Flutter which helps deal with the bloc library.

Unfortunately the analyzer API seems pretty work on progress and I wasn't able to find too much information about it online. I got a version working, but it is very inefficient.
Basically, what I need to do is analyze the whole project to find relevant subclasses of "Bloc" and a few other things which need access to the whole project.

I found "getResolvedLibrary" to resolve the whole library but it didn't seem to work. Therefore I resorted to analyzing every file in the project - every time a request comes in.

for (var path in request.result.session.analysisContext.contextRoot.analyzedFiles().where((it) => it.endsWith(".dart"))) {
  var unitElement = await request.result.session.getResolvedUnit(path);
  if (!unitElement.isPart) {
    logger.log('Analyzing $path ....');
    unitElement.libraryElement.accept(blocStateCollector);
  }
}

(https://github.com/Norbert515/bloc_builder_host_package/blob/master/tools/analyzer_plugin/bin/event_navigation.dart#L64-L71 - sorry about the messy code)

This totally worked for small projects with few files but failed on large ones.

I was hoping that "getResolvedUnit" did some internal caching - but I'm very unsure about that.

My question is, how does one go about analyzing the whole project? Are there any resources besides https://github.com/dart-lang/sdk/blob/master/pkg/analyzer_plugin/doc/tutorial/tutorial.md that could help with this?

I feel like there is a lot of power in analyzer plugins and I'd love to contribute in some way!


Thanks,
Norbert

--
You received this message because you are subscribed to the Google Groups "Dart Analyzer Discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to analyzer...@dartlang.org.