New developer(s) required for PHPstorm SilverStripe plugin.

[Matthew Hailwood]

Hey Folks, 

There was a question asked on here about what IDE people use for their development and the clear majority was PHPstorm - This is great but what also became apparent is the number of issues the only SilverStripe plugin for PHPstorm has.

TLDR;

Marcus Dalgren (the original developer) chimed in and the effective answer was that he doesn't have the time to maintain it - nor does he work with SilverStripe anymore so he's looking for someone to take over the plugin.

The biggest caveat of course is that the plugins are written in Java and we're all PHP developers!

Things to Know

• It doesn't cost to develop them - You can use the free community edition of Jetbrains IDEA
  
• Jetbrains have pretty good documentation and a tutorial on creating a custom language plugin and it's not too difficult (I don't know Java yet I still managed to follow it through).
  
• The "difficult" part of the plugins is writing the Jflex .flex file which is a bit like regex used to generate the parser (https://github.com/raket/idea-silverstripe/blob/master/src/com/raket/silverstripe/SilverStripe.flex)
• We've got a working shell for the plugin that is up to date with the current structure of the plugins, and has all depreciation issues resolved that we could build off (it just doesn't have a working Jflex setup) 
• It has been suggested to post on bountysource for this project so the community can support the maintenance of the plugin - given the usefulness of the plugin to us it sounds like a worthwhile investment.
• David Alexander has suggest he would be interested in helping out in a couple of months, so that's a start on our maintenance team.

Don't have enough time to maintain it?

No problem, I wouldn't mind maintaining the repository - handling pull requests, bug reports, and fixing what I can. We just need help to get the Jflex syntax right.

Ideally we would like to build a team of developers on it rather than just one or two devs to avoid this situation in the future.

--

So if anyone thinks they could help out, or knows someone who could let's get some buzz going and make developing in our most popular IDE as awesome as it can be.

[Marcus Dalgren]

Hi,

I'd love to help out as much as possible answering questions etc. Getting the Jflex stuff to work in the latest IDEA has proven to be somewhat of a beast.
You need to install the as yet unreleased version 1.3.1 of Grammar kit in order to be able to run the task that generates the lexer file without errors.
I fixed two bugs yesterday but as I said before I don't really have the time to spend on it anymore.

Unfortunately one of the things I have forgotten is the Jflex syntax. :(
I read that file today and while I do get the gist of it I can honestly say that I don't remember the intricacies of it.
That's a part you really have to get right bc it can end up in infinite loops if you write it wrong.

I'm curious as to why you started over in a new repo and copied some of the stuff instead of forking the original repo and continuing from there?
Running in the latest version of IDEA only two deprecation issues showed up and one of them is an easy fix and the other one can be removed I think.

[Anselm Christophersen]

Matthew, I’d also suggest to go with Marcus’ module, as this is the one most of us are using, and it’s working fairly well - what were your issues with it?

--
You received this message because you are subscribed to the Google Groups "SilverStripe Core Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to silverstripe-d...@googlegroups.com.
To post to this group, send email to silverst...@googlegroups.com.
Visit this group at https://groups.google.com/group/silverstripe-dev.
For more options, visit https://groups.google.com/d/optout.

[Matthew Hailwood]

Hey Marcus,

Your help would be invaluable and I'm happy for us to continue using your module - honestly I only started the other repo so I could run through the tutorial (as I noticed that's how you had started your version based on the example content when editing colour schemes) as I'm not a Java developer I kinda took a look at your repo and went er... what? (though after running through the tutorial I now know enough to work with it)..

It's more than likely that you had already fixed the depreciation errors that were present in the tutorial and I made the (incorrect) assumption that based on the last update age that those issues would not have been around when you were actively working on the plugin.

Do you remember if there was anywhere that you were able to get help with Jflex? Given that the officially mentioned support channels aren't exactly overflowing with activity! or were you able to just work it out?

Side note - I noticed you did the update (thanks for those :)) in the phpstorm-61 branch which "This branch is 46 commits ahead, 20 commits behind master." What's the deal with the branches?

[Marcus Dalgren]

Yeah so the branches are a mess.
Master is old and not in use. Just like you I learned everything at once (Jflex, java etc.) so the first version of the parser was a total mess.
Brittle as all hell and I felt I didn't really have any control over it (fixed one thing another one broke). So I started a rewrite of the parser in a separate branch while also making releases in the master branch.
I ended up copying stuff over in some really stupid ways so even though their history is much closer than implied by the git message they can't really be merged since they have alot of the same commits but with different SHA:s. :(
Considering the state it's in I'd need to force push phpstorm-61 as master but I haven't done that since I'm not sure if that would totally break stuff for someone.

For help learning this stuff I had two primary sources, the tutorials you found (which are kindof old now) and the handle bars plugin by dmarcotte (https://github.com/dmarcotte/idea-handlebars).
There's also the Lua plugin I mention in the readme (https://github.com/juzna/intellij-latte) that I learned some parser stuff from.
I had a look in the forums yesterday and the activity levels seem to be alot lower than when I was working on the plugin unfortunately.
During the time I spent on it you could often get answers from the Jetbrains devs about stuff which was a huge help.
The later versions of Intellij IDEA have a built in decompiler so now you can actually read the source from PHPStorm but before that all the PHP stuff was just wild guesses and trying to figure out how the Symfony2/Twig plugin does things.
Now you can at least read the source but figuring out how the more advanced stuff works is really hard and it was one of the reasons development kind of died off.

I do remember reading some Jflex tutorial about how that stuff works but can't remember where I found it. Actually it might have been this: http://jflex.de/manual.html#Example
The basics of it are that you have different states starting with YYINITIAL.
YYINITIAL will match either <% or $ or {.
When it matches <% it takes two steps back (yypushback(2)) and then switches state to SS_BLOCK_START which starts at line 141.
Everything in that section can be matched when the lexer is in the SS_BLOCK_START state.
Since we moved everything back two steps it will now read <% again but since it's in SS_BLOCK_START it will now match the token definition which is
SS_BLOCK_START= <% on line 66.
Reading this code again after about a year and a half I realize that one of the confusing things about this is that I sometimes use the same name for a state and for a token definition.
That's something I'd really change since it's confusing enough as it is.

Anyways, what you need to do is threefold.
Write token definitions which are either a somewhat limited form of regex or a literal string (line 55 to 91).
Write state definitions for the lexer (line 93-111).
Write the actual states (113-302) and that's the tricky part.
The states are a stack that you can either push a state onto or pop a state off of returning you to the previous state.

An easy example would be <% else %>.
<% will first put you in the SS_BLOCK_START state.
From there it first matches <% again and now matches SS_START_KEYWORD and returns a token type for that.
Next it matches else as SS_ELSE_KEYWORD and returns a token type for that.
And last but not least it matches SS_BLOCK_END for %> and returns the token type for that.
SS_BLOCK_END also pops the current state (SS_BLOCK_START) off the stack and puts you back in YYINITIAL.
Please note that I excluded the white space matches.
If you for some reason write code that fails to return to the initial state then you're screwed and it'll crash the IDE running the plugin.

There's a plugin called PsiViewer which you can install which let's you inspect the results.
In PsiViewer you'll see the whole token tree which will help you see if you got the results you wanted/expected.
It also helps to find the name of tokens in PHP or yaml when you need to work with those.

This got really long and there's more to discuss with those who are interested.
Should we discuss the more complicated technical stuff somewhere else?

[Matthew Hailwood]

Perhaps discuss in a github issue that we link to from here?

Yes I ended up referring to the handlebars plugin quite a bit as well - 

Most of what you've said above rings a bell from what I understand so that's good to know.

Yeah I was using the Psi Viewer it's super useful to see how it's actually breaking down the tokens if something isn't matching up.

Would you like me to to take a shot at forking the plugin,moving the current "master" to old-master so anyone who is using the current master can refer to that instead of a total wipe and then implementing git flow to manage the branches?

Seems a perfect fit for the plugin.

[David Alexander]

Just wondering out loud here:

(1) Since a well functioning SS plugin aids in the adoption of SS, perhaps SS, the "company", has an interest in the plugin's development and has some ideas on how they'd like to see this move forward ?
Maybe a repo silverstripe/phpstorm-plugin ? And, perhaps plugins for other major IDEs and editors as a longer term plan ?

(2) As part of working on the plugin, we should document its structure to make it easier for others to get started contributing.

(3) I am completely happy with Marcus continuing with his organic brain dump here...get it all written down, even if unstructured :)

[Matthew Hailwood]

The general stance from popular frameworks seems to be that they don't have enough resources to support plugins for PHPstorm, Netbeans, Sublime Text, Atom... You get the idea - but PHPstorm may be an exception here since we have a clear majority of users.

And we can't swing it the other way (getting the IDE vendors to support it) as Jetbrains basically say use this plugin ;)

Good call on the documenting it's structure.

I think (assumptions... need to stop those) Marcus want's to keep the technical details separate from this thread so this can act more as a recruiting option (we don't want people to see a 40 message long thread that's all technical details and assume that we must have a ton of people already).

[Patrick Nelson]

Curious: Has anyone picked this up yet? It stinks that we have an open source project like this that people are making heavy use of but nobody to help maintain it. I guess you don't really notice it until you find bugs (or features) and it affects you. Just wondering. I think part of this is that we're all at least PHP developer but maybe not necessarily Java developers 🤔

--
You received this message because you are subscribed to the Google Groups "SilverStripe Core Development" group.

To unsubscribe from this group and stop receiving emails from it, send an email to silverstripe-dev+unsubscribe@googlegroups.com.
To post to this group, send email to silverstripe-dev@googlegroups.com.

[David Alexander]

I would like to get into this. Time is the enemy here though!

I will make a start reading the documentation Matthew posted...

[Matthew Hailwood]

The big thing to understand is the Jflex syntax which is here https://github.com/raket/idea-silverstripe/blob/master/src/com/raket/silverstripe/SilverStripe.flex

I would suggest that to get into it attempt to just update the demo text for code colours to be actual SS template code which will tell you that you're compiling fine.

That's stored here https://github.com/raket/idea-silverstripe/blob/bcafb02a5f7efe59c785ac9f19a1a046769afb3c/src/com/raket/silverstripe/editor/colors/SilverStripeColorSettingsPage.java#L41

After that I would just into trying to get <% include TemplateName Var=$Val %> to be recognized as valid to introduce yourself to the Jflex syntax.

(can ya tell I've put a bit of thought into this?)

[David Alexander]

(can ya tell I've put a bit of thought into this?)

...and you are keen to get work on it moving along! It would be good to get a few of us working on it semi-regularly.

I had a quick look over all of the code and the Java used doesn't appear to use any really complex features of the language, although I will need to remind myself how Java handles events.

I haven't tried editing or compiling yet...I'll try to get to this during the week (small steps with small, semi-fluid deadlines work best for me at the moment).

To properly maintain it, and add new features, though, we'll have to have a pretty complete understanding of the entire plugin...which will take many small steps. :)

[Marcus Dalgren]

Yeah the code isn't all that advanced.
The trickiest parts (as I remember them) are the flex file and figuring out how stuff works.
When I built the plugin there was a basic tutorial available which kind of worked and I got alot of help from the Handlebars plugin author Daniel Marcotte.

Compiling the plugin right now throws alot of deprecation warnings but deprecations generally don't point to their replacements so you basically have to go digging.
IDEA at least has decompiling nowadays so you can look at the source and there was a new library available that seems to be the source annotations för the PHP OpenAPI.
I would check their docs again and see if they've been updated/improved and also check their forums. When I was working on it the forum was quite active and the devs did answer questions from time to time.

You guys can of course ask me questions about it as well and I'll answer them as quickly as I can.
Like I've said before I'm willing to either hand the plugin over or add more contributors to the project.
It seems that I can add developers on the Jetbrains page as well so we could start it out as a joint venture and then I can either pull out or rejoin again if we get a team going.
How does that sound?

[Matthew Hailwood]

Hey Marcus,

The latest update to the plugin you've pushed is causing an IDE exception (java.lang.IllegalStateException: Recursive runForEachCaret invocations are not allowed) when you hit enter inside of <% loop $Children %><% end_loop %>

I went to take a look but noticed you haven't pushed the latest code up to the git repo; Can you please commit the latest code and I'll take a look this evening to see if I can fix it?

Cheers,

[Marcus Dalgren]

I've pushed the latest.
However all I did was to comment out the code that made the $ sign.
It seems that compiling against a newer version of PHPStorm has introduced new behavior. :(
Could you open an issue about this on Github so we don't have to have the discussion here?