What are the performance prospects?

[Jo D]

Hi all,

we are currently evaluating Checker Framework for our software development, and given that the performance has been much slower than we expected, we'd like to know what improvements are likely to happen mid-term, and where the limits are.

DETAILS

We are trying to apply the UIEffect checker to a legacy Swing application, which weighs in at roughly 180,000 LoC.

There are roughly 500 application classes that are subclasses of JComponent, JWindow, or JDialog.

javac time without Checker Framework: ca. 35 seconds

javac time with Checker Framework and UIEffect: ca. 10 minutes

That's a factor 20 on the compilation time, and 10 minutes means some frying pan/fire choices regarding the place where Checker Framework is to be run during the development cycle.

QUESTION 1: How much of the slowdown is attributable to Checker Framework itself, and how much is attributable to the UIEffects checker?

I already know that it doesn't make much of a difference if I have (a) just UIEffects or (b) Nullness. Nullness seems to be measurably slower but not significantly.

QUESTION 2: Does the number of errors found affect the running time?

(I.e. can we expect the times to go down substantially if we fix the errors?)

I expect the answer to be No, because the the times do not vary significantly between Nullness (500 errors) and UIEffects (10,000 errors), but it would be nice to have a confirmation or a counter-observation. (I plan to get the running time when both checkers are active.)

QUESTION 3: What are the basic reasons for the long running time?

I would have expected Framework Checker to be as fast as javac itself to either pick the relevant information up, or just recompile everything and collect the data that it needs, but obviously that's not the case.

So what are the challenges that CF must overcome?

Or is it just a matter of optimizations not yet applied?

QUESTION 4: Are there plans for improving the performance?

I'm asking this because I have to plan how to use CF in our projects in the future, and expectations about the future evolution obviously play a role because using CF is going to be a big investment of our time.

QUESTION 5: What are the reasons why performance improvements haven't happened yet?

Could be "it just takes time", or lack of manpower/experts, or lack of funding.

I might be able to convince my employer to help if everything checks out (or I might not, I just want to check if there's an opportunity).

Regards,

Jo

[Michael Ernst]

Dear Jo,

Thanks for getting in touch.  I'm sorry about your frustration that the Checker Framework is slow.  That is the worst thing about the Checker Framework, in my opinion.

> QUESTION 1: How much of the slowdown is attributable to Checker Framework itself, and how much is attributable to the UIEffects checker?
> I already know that it doesn't make much of a difference if I have (a) just UIEffects or (b) Nullness. Nullness seems to be measurably slower but not significantly.

I would have suggested doing the same experiment, but using the Tainting Checker which is the most lightweight type-checker distributed with the Checker Framework.
The difference between no Checker Framework and the Tainting Checker is due to the Checker Framework.
The difference between the Tainting Checker and some other checker is due to the other checker.

> QUESTION 2: Does the number of errors found affect the running time?
> (I.e. can we expect the times to go down substantially if we fix the errors?)
> I expect the answer to be No, because the the times do not vary significantly between Nullness (500 errors) and UIEffects (10,000 errors), but it would be nice to have a confirmation or a counter-observation. (I plan to get the running time when both checkers are active.)

No.  The Checker Framework does the same checks regardless.  Eliminating errors will save only a tiny amount of time spent formatting and outputting the errors.

> QUESTION 3: What are the basic reasons for the long running time?
> I would have expected Framework Checker to be as fast as javac itself to either pick the relevant information up, or just recompile everything and collect the data that it needs, but obviously that's not the case.
> So what are the challenges that CF must overcome?
> Or is it just a matter of optimizations not yet applied?

Here are a few answers; others might chime in with reasons that I overlooked.

 * It has to do all the same work as the compiler does, such as resolving overloading and overriding, inferring generics, and type-checking.

 * Its analysis is general; it interprets user-defined type systems whereas javac hard-codes one type system and integrates it with other processing.

 * Its analyses are much richer. javac can take certain shortcuts that are not correct for all possible type systems.

 * It builds a control flow graph and performs a fixpoint analysis on it, which javac does not do.

 * It manipulates multiple representations of data, including javac’s internal representation, source code, control flow graph, and its own internal representation. These transformations and lookups take time.

 * It switches between dataflow analysis and type analysis, and this sometimes causes it to redo work.

 * When running a compound or aggregate checker, it computes the control flow graph multiple times, and it makes multiple passes over the program rather than just one.

> QUESTION 4: Are there plans for improving the performance?
> I'm asking this because I have to plan how to use CF in our projects in the future, and expectations about the future evolution obviously play a role because using CF is going to be a big investment of our time.

We would like to improve performance, but we don't have a definite timeline.  We tend to prioritize corerctness over performance.

> QUESTION 5: What are the reasons why performance improvements haven't happened yet?
> Could be "it just takes time", or lack of manpower/experts, or lack of funding.
> I might be able to convince my employer to help if everything checks out (or I might not, I just want to check if there's an opportunity).

As you guessed, the reason is resources:  we don't have the time and manpower.  We would welcome help and contributions!

                    -Mike

--

---
You received this message because you are subscribed to the Google Groups "Checker Framework discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to checker-framework-...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/checker-framework-discuss/16267088-aaca-4fbd-beef-066d100cca28n%40googlegroups.com.

[Michael Ernst]

Jo-

To improve performance, the most important thing you can do is incremental compilation.  When compiling just a few source files (the size of a typical edit or commit), you won't notice the slowdown.  If you compile all files in a large project, you will definitely notice a slowdown!  Are you able to structure your build system to make compiling all files rare, such as by declaring dependencies and using caching?

Mike

[Manu Sridharan]

> QUESTION 5: What are the reasons why performance improvements haven't happened yet?
> Could be "it just takes time", or lack of manpower/experts, or lack of funding.
> I might be able to convince my employer to help if everything checks out (or I might not, I just want to check if there's an opportunity).

As you guessed, the reason is resources:  we don't have the time and manpower.  We would welcome help and contributions!

I am happy to try to help out / advise on any efforts around improving Checker Framework performance.  (Unfortunately I do not have time to do much deep implementation work myself at the moment.)  There are some techniques we used in NullAway (basically, laziness and caching) that in principle could help speed up the Checker Framework as well.  Achieving these changes will likely require a strong understanding of Checker Framework internals, but that could be a great learning experience for someone who is interested and has the time.

Best,

Manu 

To view this discussion on the web visit https://groups.google.com/d/msgid/checker-framework-discuss/CAAJCdQTh%2BHKYEvJa76jkg3f7JbWi-kAmkjeRe3NygzxdCOZXWQ%40mail.gmail.com.

[Jo D]

Thanks for all the feedback!

I might be able to convince my employer to sink some resources into helping out, but unfortunately not for the kind of digging deep into internals like optimizing CF better would require.

I should have opportunities to send members of our trainee program in CF's direction though, particularly those with an interest in tools/compilers who might find CF a good component in their CV. Be prepared for people with a potential but who don't yet have experience with large, complex frameworks like CF.

BTW are there any LoC counts for Checker Framework or any checkers? I know it's a terrible metric, but it's easily gathered and understood, and good enough for order-of-magnitude laddering, so it would help me making recommendations to trainees.

For now, I'm back to getting some more performance figures out of CF (thanks for the tips!), and preparing a results presentation.

Regards,

Jo

[Jo D]

Here's a finding: Checker Framework needs a lot more memory than javac. I believe the examples in the manual should include some Xmx settings, just to warn that people should check the memory usage.

Here's what you need to say for maven-compiler-plugin:

<build>
    <plugins>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-compiler-plugin</artifactId>
            <configuration>
                <maxmem>512m</maxmem>
            </configuration>
        </plugin>
    </plugins>
</build>

For our project, it turned out that the default max heap (256 MB) gave these results:

- Used live (post-GC) heap usage started at ~180 MB, slightly growing to ~245 MB.

- GC took about 80% of CPU time, slowly growing to 90%.

- I.e. RAM was barely sufficient and the build just managed to scrape by (90% GC is pretty bad but it could have been 99%...).

- This was with the Tainting checker, which took almost 20 minutes (so it's actually slower than nullness and uieffect checker - no idea how that happened).

With 512 MB, I got this:

- It was enough for the Tainting checker.

- Nullness was almost content with that, it had a handful of excursions of 60-80% GC over three periods of less than 10 seconds.

- Nullness checker now took around 13 minutes this time, despite having more memory - not sure if that's significant, compared to the ~10 minutes I observed earlier.

- UIEffect took about 3:30 - MUCH less than the 9:30 I observed before. Given Nullness took somewhat longer, this is even more puzzling. I had some @UIType annotations added this time, some of them strategic, so maybe that's the cause - I suspect that an annotation in the right place can shortcut a lot of the checker's work, but I don't know how to validate such an assumption.

- Nullness combined with UIEffect took 21minutes. It started with short-lived GC excursions like Nullness alone, but got more and more GC spikes over time. The number of spikes increased significantly over time, so 512 MB was almost enough but not quite.

1 GB of RAM:

- I checked just the nullness+uieffect combination.

- Java wouldn't allocate the full amount of RAM until pretty late in the process, and it had no significant GC overhead anymore, so this was really enough.

- Running time was still 16 minutes (which looks like the sum of what the two checkers need individually).

Overall observations:

- Visualvm reported 10% of overall CPU usage, Task Manager 16%.

- This is on a CPU with 6 cores and 12 "logical processors" (as reported by Task Manager).

- I am aware that graph algorithms like those typical for type checking aren't easily parallelized, but at least for us, the difference would be significant.

On Friday, December 25, 2020 at 6:56:48 PM UTC+1 msri...@gmail.com wrote:

[Michael Ernst]

> BTW are there any LoC counts for Checker Framework or any checkers? I know it's a terrible metric, but it's easily gathered and understood, and good enough for order-of-magnitude laddering, so it would help me making recommendations to trainees.

Here are line counts, excluding tests.

                         Files     Lines   Blanks  Comments     Code
──────────────────────────────────────────────────────────
Overall                   3260    262686    32657     59116   170913
  checker                  350     37013     3431     10835    22747
    builder                  3        82        6        42       34
    calledmethods           13      1583      121       540      922
    compilermsgs             6       106       12        28       66
    fenum                   22       744       72       231      441
    formatter               13      1977      175       615     1187
    guieffect               14      1564      127       426     1011
    i18n                    10       303       32        73      198
    i18nformatter           15      2063      182       440     1441
    index                   65      9427      844      2742     5841
    initialization          10      2128      183       609     1336
    interning               13      1510      155       410      945
    lock                    20      3157      347       813     1997
    nullness                34      4834      449      1528     2857
    optional                 5       488       44       110      334
    propkey                  5       328       41        77      210
    regex                   10      1405      117       459      829
    signature               26      1016       78       376      562
    signedness              13      1690      175       528      987
    tainting                 5       111       12        36       63
    units                   48      2497      259       752     1486
  dataflow                 162     20624     2529      5010    13085
  framework                337     91536     9079     25744    56713
  javacutil                 25      9345      787      3015     5543

To view this discussion on the web visit https://groups.google.com/d/msgid/checker-framework-discuss/354c3368-7132-49f8-a9a7-a96f5d9ca642n%40googlegroups.com.

[Durchholz, Joachim]

Thanks!

Those LoC counts will be useful.

Sensitivity: C2 Internal

The content of this e-mail is intended only for the confidential use of the person addressed.
If you are not the intended recipient, please notify the sender and delete this e-mail immediately.
Thank you.

[Michael Ernst]

I should have broken out some checkers that are in the framework project.  The following chart does so.

Mike

    aliasing                 9       775       64       221      490
    initializedfields        7       440       45       115      280
    reflection              21      2047      196       474     1377
    returnsreceiver          7       388       36       121      231
    subtyping                5       277       29        75      173
    value                   39     10084      744      2089     7251

  javacutil                 25      9345      787      3015     5543

[Michael Ernst]

Jo-

Here's a finding: Checker Framework needs a lot more memory than javac. I believe the examples in the manual should include some Xmx settings, just to warn that people should check the memory usage.

Thanks for the reminder that a heap that is too small may dramatically reduce performance.

Rather than add Xmx settings to the manual, I made a different change:  the Checker Framework warns the user if it is thrashing in garbage collection.  The code is in https://github.com/typetools/checker-framework/commit/cb082cb50b848b4880207ceacd91c12e00572354 .

Mike

[Durchholz, Joachim]

I like the overall idea very much.

Just FYI: We have a similar mechanism in the product I’m working on in may day job; it’s starting a background thread, runs every xx seconds, and once it detects the condition, it emits a warning and stops the thread.

 

From: Michael Ernst <mer...@cs.washington.edu>
Sent: Tuesday, 5 January 2021 23:17
To: Durchholz, Joachim <Joachim....@six-group.com>
Cc: Checker Framework discussion <checker-fram...@googlegroups.com>
Subject: [External Sender] Re: What are the performance prospects?

 

Jo-

 

Mike

 

Sensitivity: C2 Internal