REPL painfully slow (towards unusable)

[moda...@gmail.com]

Hello!

I'm using Cursive 1.4.3-2016.3 and IntellJ IDEA 2016.3.4

I am regularly experiencing extremely slow REPL when I'm working with some (simple forms): editing is just not workable, as the whole thing freezes for long moment, forcing me to type in blindly. 

Until then, I used to get around this by printing from my source code, imitate the REPL usage to only running some forms --> not too productive.

Lately I've upgraded my MPB mid-2012 to a MBP 16Mb RAM late 2016: same behavior

From what I saw on this list, similar behavior have been experienced, and apparently fixed. Am I missing something here? IntelliJ says all my software including plugins are up to date. 

I'd appreciate any help on this,

Kind regards,

PS. the forms I'm using in the REPL are very simple:

(let [{:keys [:decl :expr]} m
      sig (:def decl)
      expression (expression! m)
      _ (swap! defs assoc (first sig) {:instruction expression :params (vec (rest sig))})
      return (return!)
      _    (println (:g body) (:root body ) (:sink body))
      body (convert (:body decl) dag)]
  ()
  
   )

[Ertuğrul Çetin]

Yes also I'm having this issue as well. It is not bearable.

[Colin Fleming]

Hi there,

Yes, recently I fixed a couple of memory leaks in the REPL client, but there may be more. If your problem is the same as Ertuğrul’s, it does look like a memory leak. Could you check this using the instructions in this comment, and if that looks like the problem and you are able to do so, could you share a heap dump as described in this comment?

Cheers,

Colin

--
You received this message because you are subscribed to the Google Groups "Cursive" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cursive+u...@googlegroups.com.
To post to this group, send email to cur...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cursive/91118bb4-d75c-4fae-9460-2268d43483ef%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Kenny Williams]

I also have this problem and it is incredibly painful. I emailed Colin a while back with a thread dump. The email was titled "REPL Latency".

[Colin Fleming]

Hi Kenny,

Does your problem look like a memory issue as described in that github thread?

Cheers,

Colin

[Kenny Williams]

No. Memory does not increase and the problem occurs from the very beginning. The symptoms get worse as the amount of text you are editing in the REPL pane grows. 

[moda...@gmail.com]

Hi Colin

So I've checked the memory consumption in the IDE during these freeze situation: all seems normal (~100Mb used heap on 450Mb allocated).

From the CPU point, things are different. I've got one Java process 100% and another one (IntelliJ IDEA) about 50%.

The Java process taking 100% CPU is the Clojure execution of a generated script, which IMO handles the NREPL side. I am not an expert with all this plumbing and am strapped for time,  so forgive my imprecisions .. 

[moda...@gmail.com]

I was wondering if anybody had an idea what could happen.

Unfortunately, I've purchased my license prior to hitting this problem :/

[Colin Fleming]

Sorry, I’m going to try to look at this today. However there seem to be several unrelated problems. One appears to be a memory leak on the client side, yours appears to be something on the server side of the REPL connection, and Kenny’s is a CPU issue of some kind. I’m going to look at Kenny’s since I have some dumps to investigate there, and I’m waiting on some memory dumps for the client side issue. It seems odd that all of these should have started at the same time, so I’m hoping that they’re related in some way and fixing one of them will help the others too.

I’m hoping to be able to fix this problem shortly, but if it’s making Cursive unusable for you and you would like a refund, let me know.

[Colin Fleming]

I’ve updated the issue over here. I can’t reproduce this issue but I have a working theory about what’s causing it. Fixing it will require some changes to the REPL completion code which will hopefully help - they’ll probably take a day or two to implement and then I’ll have a dev build to test.

To view this discussion on the web visit https://groups.google.com/d/msgid/cursive/DC244870-8B40-4E19-9BF2-758A6CBD3274%40cursive-ide.com.

[Jan Sulmont]

Hello Colin

Thanks for looking into this — as far as I could tell (monitoring OS X and the memory gauge within Intellij) there were no memory leak.

As I said, it is the REPL process which is 100% CPU. This behavior is triggered as soon as I start editing within the REPL edit pane.

It may very well due to how completion is done, I can’t tell for sure:

• It doesn’t matter which characters are entered, nor whether these characters are entered within a form, or outside the form being edited in the REPL
• The problems starts immediately after editing, regardless the REPL was freshly started or not. 

And I’d rather this fixed and me happily keeping using Cursive :D

Let me know if/how I can help

Cheers

You received this message because you are subscribed to a topic in the Google Groups "Cursive" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/cursive/UL6ICi4leV8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to cursive+u...@googlegroups.com.

To post to this group, send email to cur...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/cursive/410F1DE8-2366-4654-800D-5B82E830F662%40cursive-ide.com.

[Colin Fleming]

Right, I believe that the completion code has a design flaw where it holds a read lock while it’s doing network access, and that causes the EDT to be blocked trying to obtain a write lock. If your REPL client process is pegged for some reason and that is causing the completions to come back slowly, that would explain why your UI is locking.

However I haven’t seen anything to explain why your REPL process should be at 100% CPU, what I have seen is all on the REPL client side. Is it possible that is a problem in your application? If you can resolve that, Cursive should at least work better. That’s only masking the problem of course, but it might make your life more tolerable in the meantime.

To view this discussion on the web visit https://groups.google.com/d/msgid/cursive/1F0053BF-6174-4AA9-BCA6-56F59D1956B8%40gmail.com.

[Jan Sulmont]

A problem in my application? Of what sort?  “lein check” is happy and my code does exactly what it’s supposed to do … 

I wouldn’t call my project file complex (although I’m not an expert). I am happy too send it to you directly if you give me an email

Cheers

J

To view this discussion on the web visit https://groups.google.com/d/msgid/cursive/264A52B3-C345-46A3-BF76-2F6D694CAC41%40cursive-ide.com.

[Colin Fleming]

I was wondering if something in your app might be pegging the CPU at 100%. It would be great to see the code if it’s not sensitive - if you could zip it up and send it to cur...@cursive-ide.com with some instructions on how to reproduce, I’ll take a look tomorrow morning.

Cheers,

Colin

To view this discussion on the web visit https://groups.google.com/d/msgid/cursive/71EDD020-6751-471C-BD23-83DF0133280E%40gmail.com.

[Timothy Pratley]

Tip: check your ~/.lein/profile.clj
Remove all user plugins and see if that contributes.
I've run into some incompatibilities that completely break Cursive's
REPL and so disabled those plugins.
(sorry if you already tried this, just trying to help :) :))

> https://groups.google.com/d/msgid/cursive/9055B2C9-F67B-457A-AF68-D7D3A8E2E35D%40cursive-ide.com.

[Colin Fleming]

I’m working on a fix for this problem. It’s a pretty fundamental change to the way REPL completion currently works so I don’t want to mix it in with the current EAP changes, which I held up for this thinking it might be a simple fix. I’m going to release the current EAP now, and I’ll hopefully have a dev build with the new changes either this afternoon or Monday for anyone affected by this to test.

> To view this discussion on the web visit https://groups.google.com/d/msgid/cursive/CAANea9bqyaD36n%2BFZyVujmwwcHHpyr9ubxC_hydZkHpcSEum6w%40mail.gmail.com.

[Colin Fleming]

Hi everyone,

I commented on the issue for this problem: 

There's a dev build up now on the plugin repo site. Try it out by going to the site, selecting the dev channel, and downloading the 1.5.0-eap2 build corresponding to your IntelliJ version. Then you can install it using Settings->Plugins->Install from disk.... I'm hopeful that this will fix these slowdown issues.

Note that this is a dev build, I've given it some fairly perfunctory testing but the REPL completion might be buggy or incomplete. I'd love to know about it if that turns out to be the case.

Cheers,

Colin

To view this discussion on the web visit https://groups.google.com/d/msgid/cursive/B6D2CE5A-D421-49F9-B7F6-69EBA85B587F%40cursive-ide.com.

[Luke Burton]

Out of interest, what causes this bug? I spend many hours per day using Cursive and I've never encountered this issue, nor have any of my colleagues who use Cursive …

To view this discussion on the web visit https://groups.google.com/d/msgid/cursive/289352B7-CC1B-4143-BE15-77F7D732DFBE%40cursive-ide.com.

[Colin Fleming]

So, I don’t really know, and I’m not 100% sure that this build will fix the issue. I’ve never experienced it, and even though there have been a few reports recently, it’s still only 5 or 6 users out of ~15k or so, so I’m not sure why it should affect them and not others. I’m basing my working theory for this build off the thread dumps that Kenny sent me a while back.

An editor like Emacs gets all its editor functionality (doc, navigation etc) from the REPL via introspection of a live system. Crucially, this happens on demand when the user asks for it by invoking the appropriate action. IntelliJ is designed around static analysis, and much of its great functionality relies on being able to get information from the source code continuously, because it’s constantly running inspections over the code and so on. Cursive works like this in the main editor, which is why it doesn’t need a REPL running. However when you’re working in the REPL editor Cursive works more like Emacs, and gets completions, doc and so on from the running system. Previously it tried to do both getting the information in a similar manner to how Emacs works, but that led to a really chatty network protocol which slowed things down considerably over remote connections.

I think the problem is that some of that network traffic happened when IntelliJ was trying to resolve symbols, which it does all the time. IntelliJ uses a single writer/multiple reader locking scheme, and the Cursive symbol resolution infrastructure was called with a read lock in place, because IntelliJ assumes that you’re getting the info from your local static analysis. So that network traffic happened with a read lock held, which will have caused all writes (which happen on the Swing EDT) to be blocked waiting for that lock to release. This is consistent with the UI locking that people have seen, but I cannot explain why this has started affecting people recently or why it doesn’t affect more users. Perhaps everyone will suddenly find that their REPL is much more responsive, or perhaps the problem is actually something else.

In this dev build, Cursive builds a local model of the remote system and bases completions and symbol resolution on that. This is built asynchronously and is updated on evaluations and file loads, so it’s a much more efficient network protocol and shouldn’t hold the IDE up while it’s syncing. Even if this turns out not to be the problem it’s a better model for Cursive, and will hopefully mean I can get better feature parity between the REPL editor and the normal code editors, which are currently much more powerful.

Cheers,

Colin

To view this discussion on the web visit https://groups.google.com/d/msgid/cursive/B71F1FB7-8E54-4707-89A9-33D6D5E14C3C%40icloud.com.

[Moiz S Jinia]

Awesome. And Cursive has 15k users! That's great to know.

Moiz

email to cursive+unsubscribe@googlegroups.com.

To post to this group, send email to cur...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/cursive/DC244870-8B40-4E19-9BF2-758A6CBD3274%40cursive-ide.com.
For more options, visit https://groups.google.com/d/optout.



--
You received this message because you are subscribed to a topic in the
Google Groups "Cursive" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/cursive/UL6ICi4leV8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to

cursive+unsubscribe@googlegroups.com.

To post to this group, send email to cur...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/cursive/410F1DE8-2366-4654-800D-5B82E830F662%40cursive-ide.com.
For more options, visit https://groups.google.com/d/optout.



--
You received this message because you are subscribed to the Google Groups
"Cursive" group.
To unsubscribe from this group and stop receiving emails from it, send an

email to cursive+unsubscribe@googlegroups.com.

To post to this group, send email to cur...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/cursive/1F0053BF-6174-4AA9-BCA6-56F59D1956B8%40gmail.com.
For more options, visit https://groups.google.com/d/optout.



--
You received this message because you are subscribed to a topic in the
Google Groups "Cursive" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/cursive/UL6ICi4leV8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to

cursive+unsubscribe@googlegroups.com.

To post to this group, send email to cur...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/cursive/264A52B3-C345-46A3-BF76-2F6D694CAC41%40cursive-ide.com.
For more options, visit https://groups.google.com/d/optout.



--
You received this message because you are subscribed to the Google Groups
"Cursive" group.
To unsubscribe from this group and stop receiving emails from it, send an

email to cursive+unsubscribe@googlegroups.com.

To post to this group, send email to cur...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/cursive/71EDD020-6751-471C-BD23-83DF0133280E%40gmail.com.
For more options, visit https://groups.google.com/d/optout.


--
You received this message because you are subscribed to the Google Groups
"Cursive" group.
To unsubscribe from this group and stop receiving emails from it, send an

email to cursive+unsubscribe@googlegroups.com.

To post to this group, send email to cur...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/cursive/9055B2C9-F67B-457A-AF68-D7D3A8E2E35D%40cursive-ide.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Cursive" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cursive+unsubscribe@googlegroups.com.

To post to this group, send email to cur...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cursive/CAANea9bqyaD36n%2BFZyVujmwwcHHpyr9ubxC_hydZkHpcSEum6w%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Cursive" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cursive+unsubscribe@googlegroups.com.

To post to this group, send email to cur...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cursive/B6D2CE5A-D421-49F9-B7F6-69EBA85B587F%40cursive-ide.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Cursive" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cursive+unsubscribe@googlegroups.com.

To post to this group, send email to cur...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cursive/289352B7-CC1B-4143-BE15-77F7D732DFBE%40cursive-ide.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Cursive" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cursive+unsubscribe@googlegroups.com.

To post to this group, send email to cur...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cursive/B71F1FB7-8E54-4707-89A9-33D6D5E14C3C%40icloud.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Cursive" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cursive+unsubscribe@googlegroups.com.

To post to this group, send email to cur...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/cursive/3A61604E-17D9-4663-8C26-0A400BB76ADF%40cursive-ide.com.

[Colin Fleming]

That’s a rough guess, I’d have to write some reports up, but that’s probably a good ballpark. Mostly non-commercial, but lots of commercial and personal ones too. It’s hard to know how much the non-commercial users are actually using it, though.

To view this discussion on the web visit https://groups.google.com/d/msgid/cursive/CAFoDxaM%3DtQONmeHZRfM4YZ35WKXnuCOTnvREryUAK6%3D-%2B5WbVw%40mail.gmail.com.

[Kenny Williams]

Just tried the dev build on my Mac and I am no longer experiencing the freezes. However, now I occasionally will have cases where the auto-complete dropdown appears and then goes away before I am able to select an item. The auto-complete is a lot more "flickery," if that makes sense. 

[Colin Fleming]

Hmm, that’s strange. I wonder what it is about your project that’s causing this. Can you reproduce this reliably, e.g. completing a particular type of element, or does it seem arbitrary?

If you can reproduce this, could you send me a screen recording of what you’re seeing? You can use Quicktime to do that: https://support.apple.com/kb/ph5882. If you need space to upload it I can provide some.

[Kenny Williams]

It seemed to happen at arbitrary times with the issue getting worse the more nested my forms were. Though, I just tried for 10 minutes to reproduce the problem while recording and was not able to. Maybe some process on my laptop was using a higher CPU percentage which caused a flaky dropdown. It could have been anything (though, it was consistently happening earlier this morning). In any case, it seems to have disappeared. If it occurs again I'll be sure to record it and send it your way. 

[Colin Fleming]

Ok, thanks Kenny. Definitely let me know if it reoccurs, and I’d be very interested to hear from anyone else who suffered from this problem too.

[Bill Kennedy]

Hi, Collin,

I think I’m seeing the issue (it’s a little confusing at this point whether it’s the same or different).

First, it happens on one particular project out of several projects that I work on.  It started happening roughly 6 to 8 weeks ago I guess.  Any time I would open this one project, the editor would hang for quite a while (the status bar would say “Indexing” and the spinner would be stuck).  After several minutes, the editor would recover and I could edit just fine.  Then, once a REPL was open then it would continually lag almost at every key press.

It was painful enough that I quit using IntelliJ/Cursive for that particular project.

I went ahead and gave it another try today after updating to the latest Cursive release (not the development one you mentioned).

The problem is still basically there, but in a little bit different form.  I still had the long grind on opening the project.  However, when running the REPL, it does not hang the same way it was doing before.  Now, as I am typing or editing in the REPL, the button “Interrupt Current Execution” lights up, even though I haven’t evaluated anything new to the REPL yet.  That’s what was happening before, but the editor would be stuck for 5 seconds or more per pause.  Now, the button lights up, but the editing does not hang in the same way.

Hopefully this information helps.  I don’t have anything particularly sensitive in the project, so I’ll zip it and email it to you.

Let me know if you have any further questions.

Bill Kennedy

To view this discussion on the web visit https://groups.google.com/d/msgid/cursive/E598B3F0-B63B-4D86-9369-F294615A9EFC%40cursive-ide.com.

[Colin Fleming]

Hi Bill,

Thanks for the project, I’ll try to reproduce this.

It seems like you have a couple of different issues there. A couple of questions:

1. Do you have the memory indicator enabled (Settings→Appearance & Behavior→Appearance→Show memory indicator)? Do you see memory being used and not freed?
2. When you say the editor lagged once the REPL was open, do you mean the standard code editors or the REPL editor?
3. Does the dev build help either of these problems?

Thanks for helping investigate,

Colin

To view this discussion on the web visit https://groups.google.com/d/msgid/cursive/F9B22671-D1B3-49C8-9855-389B0A538A91%40cataboom.com.

[Bill Kennedy]

Hi, Collin,

1. Memory used is staying pretty much constant (does not appear to be a memory leak).

2. I mean the REPL editor.  I don’t get the lag in the standard code editor.

3. I’ll try the dev build tomorrow and let you know.

Regards,

Bill

Bill Kennedy

bken...@cataboom.com

214.477.5719 (mobile)

To view this discussion on the web visit https://groups.google.com/d/msgid/cursive/96160F78-11F4-4029-8819-286A41C18600%40cursive-ide.com.

[Alan Thompson]

Hey Colin - I'm not sure what email client you're using, but it appears to be incompatible with the GMail email history folding.  This email thread is about 15 pages long now!

GMail seems able to fold Bill's messages fine..... not sure what the difference is.

Alan

To unsubscribe from this group and stop receiving emails from it, send an email to cursive+unsubscribe@googlegroups.com.

To post to this group, send email to cur...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cursive/E598B3F0-B63B-4D86-9369-F294615A9EFC%40cursive-ide.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Cursive" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cursive+unsubscribe@googlegroups.com.

To post to this group, send email to cur...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cursive/F9B22671-D1B3-49C8-9855-389B0A538A91%40cataboom.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Cursive" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cursive+unsubscribe@googlegroups.com.

To post to this group, send email to cur...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/cursive/96160F78-11F4-4029-8819-286A41C18600%40cursive-ide.com.

[Colin Fleming]

I’m not sure what’s going on there, I’m just using Apple Mail. It looks like they’re being folded ok on the Groups site: https://groups.google.com/d/topic/cursive/UL6ICi4leV8/discussion

To view this discussion on the web visit https://groups.google.com/d/msgid/cursive/CAN67zA2Hu3fm7A-uwiqpKOqTR34c2EhyByTrdqPffNijdJ9VVQ%40mail.gmail.com.