Splitting GF into multiple repositories

[John J. Camilleri]

The idea to split GF into multiple repositories has been floating around for a while. It seems there are no real objections, but someone needs to take the lead and decisions need to be made. Let this thread be a common discussion for these decisions. I volunteer to take the lead, and I am motivated to see this done by/during the summer.

Aarne's suggestions from a previous mail thread are pasted below:

- HS:  all code in GF/src needed for the grammar compiler and the Haskell runtime - in other words, for building the program called "gf"
- Runtime: the C runtime and the bindings to it from Java, Python, Haskell, Javascript, DotNet, ...
- RGL: the contents of GF/lib except translator/ and experimental/ - in other words, for building the standard RGL and its API documentation
- Examples: GF/examples with GF/lib/src/translator moved here
- Tools: mainly GF/src/ui and GF/src/server
- Documentation: GF/doc i.e. tutorial and reference manual, as well as the web page
The dependencies between these sets are simple enough ("A -> B" means "B depends on A")
- HS -> RGL -> Examples
- Runtime,HS -> Tools
- Documentation
However, the situation will be different if the HS runtime is deprecated and the gf shell starts to use the C runtime, as Krasimir has planned for the future. Then it would make sense to package HS (or parts of it) and Runtime together.

Given this last comment, I think that for now all the GF code (compiler, all runtimes, server) should still be kept together. In other words, we want to keep inter-repository dependencies to an absolute minimum.

The current `GF` repository is huge and some parts of it are very old. I don't think we want to necessarily copy everything forward. I suggest the following:

1. We create as many new repositories as needed (see below).
2. We copy the things we want into the new repositories.
3. The `GF` repository is frozen and becomes an archive.

The point here is that we don't have to find a place for everything immediately. Instead, we need to find a place for things which are current, and old things can go on living in the archive repo until someone decides to revive them.

So, my suggestions for the new repositories (similar to Aarne's suggestions above):

1. gf-core: compiler, shell, runtimes, server
2. gf-rgl: the RGL and its documentation
3. gf-doc: tutorials, reference manuals, website
4. gf-tools: mobile apps, Eclipse plugin, editor modes etc.
5. gf-examples

Some immediate questions:

1. Should we put all "tools" together, or should they have individual repos? This may also apply to "examples".
   
2. Currently the contents of the `GF` repository are served from www.grammaticalframework.org. How should this work in the future? Should documentation and website be in the same repo? Maybe we should have a repo for `gf-web` and the documentation moves to `gf-core/docs`?

The list above is certainly incomplete. Help me improve it!

Maybe someone with experience from other OSS projects has some ideas: what's a good way to collaborate on this suggested list of new repos and what should go in each? Email doesn't seem very effective. Perhaps a GitHub "project" (which is like a Trello board?) I wonder if we might need some rudimentary voting system too.

John

[bruno cuconato]

hello John,

I agree with your suggestions, specially with the one about not copying everything forward!

Should we put all "tools" together, or should they have individual repos? This may also apply to "examples".

I

​'d prefer to have things in separate repositories, as (at least in my experience) tools usually do not depend on each other! we'd have to have a page on the website listing them all, naturally!

about your second question: at the moment I only have a slight idea of how GF documentation is built, so I don't think I can offer a thorough opinion about organization yet. if more people also lack this knowledge it would be nice to summarize it! (for instance, the GF synopsis is most likely assembled by a custom script; are any other pages like this?)

​I do have some suggestions about documentation, though:

- I think it'd be nice to have a canonic, rolling-release GF tutorial, and an up to date reference​. currently we have the GF book, which is great but a bit out of date, the GF tutorial, the GF RGL tutorial, and even a little bit of tutorial in the RGL synopsis.

- I also miss a GF FAQ! (we could make a habit of putting those questions that pop up from time to time there)

- I don't know how the documentation is currently assembled, but it would be nice to have it exported to HTML from a simpler markup language (like markdown or org-mode), and have on-page links to edit its source (this really favours those small updates that might take a few seconds to do, but are also very minor, so they end up postponed if we do not reduce the time cost of making them). this of course goes for the tutorial, the reference manual, the quickstart, the quickrefcard...

my last point relates a little to last bullet point: should we use a static site generator like jekyll/hakyll/etc to generate the GF website? if so, which one?

btw, this might come in handy when splitting the main repository, in order to preserve history. 

​I'm glad to help if I can!​

​

-- bruno cuconato

--

---
You received this message because you are subscribed to the Google Groups "Grammatical Framework" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gf-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[John J. Camilleri]

Thanks for your reply, Bruno.

Although I might be considered a GF veteran, I still get daunted when I peek into certain parts of the GF repo! But here are some replies to your questions and suggestions.

Some documentation is generated by scripts, in particular the RGL synopsis which contains all the language paradigms is generated from GF source by lib/doc/MkSynopsis.hs

Some of the other HTML documentation files are generated from txt2tags format (t2t), for example doc/gf-editor-modes.t2t

And some other HTML pages are not generated at all, but edited manually, e.g. doc/runtime-api.html

Although I like to use site generation personally, I don't think it's a good idea for the GF project because it would mean another thing to set up, another hosting environment dependency, and another language/format for people to learn. And since we already have a big body of documentation which exists in other formats, it would mean doing lots of conversions of files etc. So while I am in favour of clearing up the documentation, I'm not really in favour of switching to a site generation system.

I like your suggestion about having on-page edit links! I guess the links can just point the GitHub's fork/edit page.

At this point I should point out that in this brave new world of split repositories, I think that the RGL documentation should live together with the RGL source code, and not in another documentation repository. This rule should probably apply elsewhere: GF language documentation should live with the compiler, same for the shell, etc. So actually it's a bit sloppy of me to just refer to "documentation" generally. Any views on this anyone?

Thanks for the link about splitting multiple folders into a new repo, I was not aware that was possible.

Finally, I have set up a GitHub "project" for managing this re-organisation. Please visit and contribute:

https://github.com/orgs/GrammaticalFramework/projects/1

​

John

[Jordi Saludes]

With the “1” at the end, it gives me a 404

Without it, it says that:

Encès 11 de maig de 2018 a 10:15:48, J. Camilleri John (jo...@johnjcamilleri.com) va escriure:

[John J. Camilleri]

Whoops, I assumed the project was public by default. It is public now, but only for viewing. If anyone wants to make changes I can add their Github username as a collaborator.

--

[bruno cuconato]

​Although I like to use site generation personally, I don't think it's a good idea for the GF project because it would mean another thing to set up, another hosting environment dependency, and another language/format for people to learn.​

​this is indeed a problem, although it can be softened by using a haskell library like hakyll, right?​

the documentation in .t2t can be converted automatically using pandoc, simple HTML pages like the runtime-api can probably be converted too! for the more complex ones, we could just continue using HTML, it is no problem for site generators.

I know this proposal is a little radical and has its downsides, but having a consistent website using just one well-known markup language should make it easier for people to contribute documentation -- and this is fundamental in order to have up to date documentation, as it is difficult to keep track of everything.

how should we go about contributing? presently I can only see the cards, not edit them. I think we should start by splitting the biggest components first: the compiler/tools, then the rgl, etc, and changing the CI accordingly.

sorry for late reply (I thought I had already replied, but the draft was sitting there, unsent!)

-- bruno cuconato

[John J. Camilleri]

Well, introducing a new library like Hakyll is exactly what I'm hoping to avoid.

I think possible solutions to this problem will become clearer after the documentation/website has actually been moved, as it will inevitably lead to some sorting and clearing out of current documentation. So my suggestion is that we postpone the discussion about site generation until after we have decided about the repository split.

Bruno, I have made you a collaborator on the GitHub project so you should be able to edit it. Anyone else is free to give me your GitHub username so that I can add you too. But if that's too much commitment, at least reply to this thread and share your feelings about the repository split. We will go forward with this, so this is your chance to make your voice heard.

John

[bruno cuconato]

ok, let's postpone it!

I'm not really sure about how that UI is to be used, so I'll write here:

- what are your thoughts on this?

​I think we should start by splitting the biggest components first: the compiler/tools, then the rgl, etc

​or should we just plan the splits and then do everything at the same time?

if you agree with starting now, can we create a new repo for one of the components?

- as the git commands used for splitting repositories (while preserving history) operate on directories, it would be nice to specify which directories of the current repo go in which new repos, and which ones are left to be archived (we should take special care not to bring along anything that can be safely archived, right?)

- two related points are continuous integration and build scripts: the former will have to be split into at least two, and I feel like there are a few redundant forms of the latter, which we should archive...

-- bruno cuconato

[Peter Ljunglöf]

Hej,

just a comment about the website/documentation: Why not use Github Pages? It supports Markdown and Jekyll out of the box, you don't have to install anything on your computer (but it makes things easier). I'm pretty sure that it can be published to an external url too, so we can keep grammaticalframework.org

/Peter

-- bruno cuconato

<Screen Shot 2018-05-11 at 10.11.36.png>
​

John

To unsubscribe from this group and stop receiving emails from it, send an email to gf-dev+un...@googlegroups.com.

[John J. Camilleri]

Remember that there is a LOT of stuff which is accessible under the grammaticalframework.org domain but which is not part of the main repository, for example all the GF user accounts. Just a random example:

http://www.grammaticalframework.org/~aarne/paris-2011.pdf

Moving to GitHub pages or similar would mean all such links suddenly stop working. Admittedly, this might be a good thing in the sense that much of it is old and outdated, on the other hand it is not good for historical things to disappear like that. One solution could be that we move everything to an archive subdomain, which in fact already exists:

http://old-darcs.grammaticalframework.org/~aarne/paris-2011.pdf

And that www.grammaticalframework.org is cleansed and only serves up-to-date documentation.

This will also mean that www.grammaticalframework.org no longer serves the contents of the main repository, so URLs like this won't exist anymore:

http://www.grammaticalframework.org/lib/src/english/CatEng.gf

But maybe it is good to move away from this model anyway. Websites are for documentation and repositories are for code, and perhaps the two shouldn't be confused.

I will emphasise here that these changes require a good deal of administration on the GF server. While I am willing to do the work itself, I would like to have the support of the community and the GF server admins before going ahead and changing things.

John

[John J. Camilleri]

TL;DR: The main upcoming split is that RGL will be get its own repository and its documentation will be improved. Self-contained tools should be put into their own repositories by their maintainers as they see fit.

So, here is a little update about the plans for the GF repository.

RGL

Rather than making many significant changes in one go, we've decided that it is more prudent to do things in stages. The first big change will be moving the RGL into its own repository, with tentative name `gf-rgl`. Moving the files themselves is easy enough, and there's already a script for doing this here. The tricky part is how this effects the build process. Building the GF runtime/compiler is one thing and compiling the RGL is another, but there should still be some convenient way of just updating and building everything without worrying too much about the details. I'll work more on this soon.

Documentation

When it comes to documentation, we want to stick to the principle that "documentation should live together with the thing it documents." So, documentation about the GF language, shell etc. should live in the main repository (aka `gf-core`), and documentation for the RGL should live in `gf-rgl`. At this point we don't see the benefit of having a separate repository just for documentation. The RGL documentation, aka "synopsis", could do with a good update, both in terms of cleaning up the code and making the interface a bit more modern. I will try to work on this as part of the repository split. Suggestions for good library documentation interfaces are welcome. At the same time I also plan to update the RGL Source Browser tool.

Website

Because there will be no dedicated documentation repository, the web pages will mainly live in `gf-core` and served directly from the repository, in the sense that pushing to the repository means updating the website. Most pages are in txt2tags format, and while it's not my first choice I see no reason to convert everything at the moment. There is however some cleanup to be done, and I have already stared this. The RGL documentation will become a special case, in that it will live in a separate repository but also be served from the GF website. Perhaps we should create the subdomain `rgl.grammaticalframework.org` (or `lib.`) for hosting the RGL documentation.

I will add here that there is much more available via www.grammaticalframework.org than is found in the GF repository. That is because our server admin has gone to some length to try and preserve old URLs and break as few links as possible during the move from darcs to git in the recent past. In addition, there is quite a lot of content under the user account urls (e.g. www.grammaticalframework.org/~john/) which we don't want to chuck out of the window. For this reason we are reluctant to change the website hosting and move to something like GitHub pages. 

Tools

GF-related tools that are more or less self-contained should generally live in their own repositories. This includes editor modes (e.g. Atom and Emacs), the new gftest tool, and even the mobile apps. In general I think it should be up to the maintainers of each tool to move and maintain these separate repositories. It is quite easy to create a new repository from a subfolder of another while keeping all the history; see here. If you want a repository under the GF organisation on GitHub, I can help you with that.

Naming

Option 1 (my current preference)

• `GF` remains as the main repository (also referred to as "core")
• `RGL` is the new repository for the RGL
• There is no dedicated archive; old stuff is simply removed from the repositories; it will always remain accessible in the git history, and tags can be used to facilitate this

Option 2 (previously proposed)

• `GF` becomes an archive which is frozen
• `gf-core` is a new repository for compiler, runtimes, etc. and webpages
• `gf-rgl` is a new repository for the RGL and its docs

Phew!

As usual, I welcome any comments to any of the above. I will interpret silence to mean agreement and/or indifference :)

Have a nice weekend everybody!

John

-- bruno cuconato

To unsubscribe from this group and stop receiving emails from it, send an email to gf-dev+un...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

---
You received this message because you are subscribed to the Google Groups "Grammatical Framework" group.

To unsubscribe from this group and stop receiving emails from it, send an email to gf-dev+un...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

---
You received this message because you are subscribed to the Google Groups "Grammatical Framework" group.

To unsubscribe from this group and stop receiving emails from it, send an email to gf-dev+un...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

---
You received this message because you are subscribed to the Google Groups "Grammatical Framework" group.

To unsubscribe from this group and stop receiving emails from it, send an email to gf-dev+un...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

---
You received this message because you are subscribed to the Google Groups "Grammatical Framework" group.

To unsubscribe from this group and stop receiving emails from it, send an email to gf-dev+un...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

---
You received this message because you are subscribed to the Google Groups "Grammatical Framework" group.

To unsubscribe from this group and stop receiving emails from it, send an email to gf-dev+un...@googlegroups.com.

[Robert Clouth]

Hey! 

Just chiming in as a total amateur trying to learn GF, but what would be reallllly nice is full autocompletion for ides. I've made a quick vscode plugin that does basic syntax highlighting, but if we put a bit more effort in together and create a gf language server, we'll be able to have autocompletion and hover documentation for RGL! I feel like I spend half the time flicking between the ide and the RGL documentation trying to find the correct function.

As a beginner this would be awesome. It could extract the documentation from the source code if it was standardised in some way.

[bruno cuconato]

hi John,

first all of all, thanks for all this!

I support basically everything you said, except I'd prefer the second naming option: the GF repo, as it stands, has ~1GB, and is very slow to clone. (one of the reasons I support the split is that the repos would get lighter). if we keep the main GF repo as is, its size will remain the same (`git gc && git count-objects -vH` shows it), we'll have to download even more data in order to clone the basic GF setup, i.e. the core and the rgl.

-- bruno cuconato

[bruno cuconato]

hi Robert,

perhaps we should create a new thread for that? I'd certainly love to have this, specially if we can use LSP and have it work with all editors.

that said, I have zero experience with LSP. I recently hacked the existing gf-mode for emacs in order to add something like hover documentation, but it only works for things defined in the current module or its abstract counterpart, so not that useful. if we had the GF compiler output the data we need in a machine-readable format it would help a lot too.

-- bruno cuconato

[John J. Camilleri]

Hi Bruno,

You are very right that the GF repo is large and slow to clone, and of course simply git removing things does not shrink anything because the history is still there. But are you suggesting that a new gf-core repository should not carry over the history from GF? I think this would be quite a shame.

A third option is that instead of git removing, we perform a multi-directory split (as you linked to here, but which I haven't tried yet). Or more practically, we nuke everything RGL-related from the history of GF but keep everything else (for archival reasons). This may already reduce the size considerably, but remains to be tested.

John

[John J. Camilleri]

Hi Robert,

Indeed this would be nice, and I can tell you that even experienced GF developers are constantly switching between text editor and RGL documentation!

I worked on a GF Eclipse Plugin a while ago which essentially tried to provide just this. It's not maintained anymore, and probably Eclipse is not that cool anymore (ahem) but it might be helpful at least for some inspiration. The LSP thing seems to be the future and it would be great to have GF support for that. Any takers? :)

But of course this definitely falls into the category of "standalone tool" and deserves it's own discussion thread/repository.

John

[Krasimir Angelov]

На пт, 15.06.2018 г., 23:07 bruno cuconato <bcc...@gmail.com> написа:

 the GF compiler output the data we need in a machine-readable format it would help a lot too.

Some support is already implemented. If you run the compiler with the --tags flag and a source file, the compiler will scan the file along with all dependencies. For every file it will dump a .gf-tags file which contains the indentifiers in that module together with the source locations. This can be used for word completions and "Go to definition" like functionality. 

[bruno cuconato]

hi John,

both ways seem bad to me: not preserving history is a no-go, but having each split repo contain all of GF's history is very troublesome too -- we'd have to download several 1GB repos instead of one.

I thought that repos like your rgl-source-browser would be automatically be made smaller by git, but apparently this is not so -- I just tried cloning it here and it was much bigger than it should've been.

maybe this might help -- the idea is to preserve the history of tracked files while scrapping the history from the ones which are removed. I just ran the commands in the second answer on your rgl-source-browser and could reduce its size to less than 1MB, but I can't seem to push it with this size... (it might be that this only works to reduce the size after cloning, which partly defeats the purpose...)

-- bruno cuconato

[bruno cuconato]

hello Krasimir,

that's good to know, thank you! when I was trying to implement this functionality I was using the show_source command in the GF shell, but its output is not a good fit for this, so I ended up using show_operations, which (as I said above) only shows opers defined in the current module and in its abstract counterpart...

how would one go about using the --tags flag, though? its not too slow to run, but then one would have to parse its output to see where the .gf-tags files were written, then parse them all to an in-memory data structure, and then read the lines in question as needed? or is there a better way?

also, what is the `indir` value supposed to mean in the files? that a thing is available in a given module but is not defined there?

-- bruno cuconato

[Krasimir Angelov]

On 16 June 2018 at 23:10, bruno cuconato <bcc...@gmail.com> wrote:

how would one go about using the --tags flag, though? its not too slow to run, but then one would have to parse its output to see where the .gf-tags files were written, then parse them all to an in-memory data structure, and then read the lines in question as needed? or is there a better way?

If the idea is to have a server which runs all the time and provides language services then maybe an even better solution is to integrate the server in the compiler then it wouldn't have to dump files on the disk.

also, what is the `indir` value supposed to mean in the files? that a thing is available in a given module but is not defined there?

Exactly, `indir` means that the name is available but defined elsewhere.

There should be other services as well, like extracting the type of something. I thought that I did that as well but it isn't. Apparently it was not needed for the Eclipse plugin.

[John J. Camilleri]

Hi Bruno,

On Sat, 16 Jun 2018 at 22:55, bruno cuconato <bcc...@gmail.com> wrote:

hi John,

both ways seem bad to me: not preserving history is a no-go, but having each split repo contain all of GF's history is very troublesome too -- we'd have to download several 1GB repos instead of one.

I thought that repos like your rgl-source-browser would be automatically be made smaller by git, but apparently this is not so -- I just tried cloning it here and it was much bigger than it should've been.

I also assumed this to be the case. This is in fact mentioned in the git documentation:

"People expect the resulting repository to be smaller than the original, but you need a few more steps to actually make it smaller, because Git tries hard not to lose your objects until you tell it to."

Anyway, it seems I managed successfully to shrink the rgl-source-browser repo while preserving the history. You are welcome to clone again and confirm! The general steps I followed (interesting ones in bold):

cp -R GF rgl-source-browser

cd rgl-source-browser

git filter-branch --prune-empty --subdirectory-filter lib/doc/browse --tag-name-filter cat -- --all master

git for-each-ref --format="%(refname)" refs/original/ | xargs -n 1 git update-ref -d

git reflog expire --expire=now --all

git gc --prune=now

git remote set-url origin "g...@github.com:GrammaticalFramework/rgl-source-browser.git"

git push --set-upstream origin master

I also had to delete the GitHub repo and recreate it; I think part of the problem is that when I created the first repo on Github I initialised it with a README, which I then had to merge with my local filtered (but not pruned) repo. Anyway, this seems like good news so hopefully the new RGL repository will also be shrunk. I don't know yet if we can also shrink a repository in place (GF) or if we will still need to create a new gf-core repo in order to bring the size down.

John

[bruno cuconato]

that's good news, John! 

You are welcome to clone again and confirm!

it works :D

​I 

 

​​

also had to delete the GitHub repo and recreate it;

​wouldn't `git push -f` do the trick?​

 

​Anyway, this seems like good news so hopefully the new RGL repository will also be shrunk.

​that would be great!​

 

​I don't know yet if we can also shrink a repository in place (GF)

​I think it would work no problem, but then anyone who tries to pull from it would have to resolve (artificial) conflicts or rebase, because history is rewritten. because of this and the fact that it would the be nice to have the original history preserved, I'd still vote to have the gf-core repository!

-- bruno cuconato

[John J. Camilleri]

So, I believe I have successfully managed to shrink gf-core and gf-rgl to under 100MB each. I have pushed them here:

https://github.com/GrammaticalFramework/gf-core (GF without /lib)

https://github.com/GrammaticalFramework/gf-rgl (just /lib)

Note that these repos are only for testing purposes! You can clone them yourselves to confirm that they are indeed smaller. It would also be great to know if the history has been correctly preserved, but that's quite hard to tell by just looking. Nevertheless, it would be good to "keep an eye out" for anything that looks weird in these repos.

If you're interested in the commands used to create these repos, see the current split script.

​I don't know yet if we can also shrink a repository in place (GF)

​I think it would work no problem, but then anyone who tries to pull from it would have to resolve (artificial) conflicts or rebase, because history is rewritten. because of this and the fact that it would the be nice to have the original history preserved, I'd still vote to have the gf-core repository!

Right. This would mean that someone's workflow would suddenly break if when they pull from GF they start getting errors about conflicts.

By creating a new gf-core (and gf-rgl), using the new repos becomes opt-in rather than enforced. That is probably a good thing. Of course people who don't opt-in will not get new updates, but at least their setup won't break.

John

[John J. Camilleri]

I have also created another version of gf-core using the BFG tool (which took seconds instead of hours). I have uploaded it here:

https://github.com/GrammaticalFramework/gf-core-bfg

It should be identical to gf-core, but this remains to be verified. This method is in theory more destructive because it only matches directory names, not repository paths. Commands can be found here.

John

[bruno cuconato]

a 72MB RGL is great stuff!

I have check the part of the history I know and things seem ok. I hear the biggest problems happen when files are renamed, but I don't recall any instances of this to check how the split handled them.

Of course people who don't opt-in will not get new updates, but at least their setup won't break.

​we could setup a github-side post-commit hook that would mirror the commit on the all-things-gf repo, but that would probably be too much work for little use.​..

 

-- bruno cuconato

[John J. Camilleri]

TL;DR: details about building and installing GF and the RGL in light of the planned repository split

I am trying to work out the details of the new build processes. Here's where I am so far (I am mostly writing this to help organise my thoughts).

Some preliminary notes:

1. "GF" refers to the compiler, shell, runtimes etc (not including the RGL), and "RGL" refer to just the RGL grammars and docs
2. GF, gf-core, and gf-rgl refer to actual repositories (current and future)
3. "building" means compiling stuff from source into an automatically created dist folder
4. "installing" means the copying files from dist to somewhere less temporary (the dist folder is an intermediate thing, and it should always be safe to remove it)

The idea with this repository split is to make GF and the RGL a bit more independent. Concretely, I think it means that the following should be true:

1. One can install and use GF without even downloading the RGL
2. One can build the RGL without needing to build GF (e.g. using a downloaded binary)
3. The details of how to build the RGL should only be specified in gf-rgl.  Currently, most of GF:Setup.hs is about building the RGL, and this should be moved to gf-rgl's own build script. (Note: there are already some RGL build scripts, notably GF:lib/src/Make.hs and GF:lib/src/Makefile. I'm not sure how up to date these are considered, but they should form the basis for the new RGL build scripts.)

Despite the above, it should be as easy as possible to get up and running with both GF and RGL, the way it is now. My current idea is as follows:

• gf-core$ cabal install will build GF (with no RGL) into gf-core/dist and then copy it to the usual install location
• gf-rgl$ make install will build the RGL into gf-rgl/dist and then copy the gfo files into GF's install location above*

In other words, GF will continue to look for compiled libraries in the same place as now, except that GF's build scripts will no longer put them there.

In this way, the binary distributions of GF can look the same, i.e. they will still include the RGL gfo files in basically the same place.

*This is dependent on a reliable way of discovering GF's install location from outside its own build script. I don't know how this might look yet but I assume some decent solution can be found.

Finally, a note about build systems. I know there are lots of build systems out there today. GF is using Cabal, but there's Ant, Grunt, Gradle, and certainly many others. I don't really know much about these, and since I want to change as little as possible at this stage, the plan for the RGL's build script will remain a combination of Haskell & Makefile (I don't think Cabal makes sense here). Improvements to this can be considered after the dust from the repository split has settled.

John

[Krasimir Angelov]

On 6 July 2018 at 08:44, John J. Camilleri <jo...@johnjcamilleri.com> wrote:

Finally, a note about build systems. I know there are lots of build systems out there today. GF is using Cabal, but there's Ant, Grunt, Gradle, and certainly many others. I don't really know much about these, and since I want to change as little as possible at this stage, the plan for the RGL's build script will remain a combination of Haskell & Makefile (I don't think Cabal makes sense here). Improvements to this can be considered after the dust from the repository split has settled.

Ant and Gradle are used mostly for Java. Grunt as far as I know is popular for JavaScript. It is better to stay away from those unless if we really use Java/JavaScript. Make is available by default on all Linux systems so I would prefer to stick to it. It is better if the build system for the RGL doesn't depend even on Haskell. In this way someone who has downloaded the GF binary could build the RGL without installing Haskell.

[John J. Camilleri]

It is better if the build system for the RGL doesn't depend even on Haskell. In this way someone who has downloaded the GF binary could build the RGL without installing Haskell.

Yes, that is a very good point. There will inevitably be some logic which probably can't be handled by a Makefile alone, and there Bash scripts seem like the obvious solution. But I'm not sure what implications this has for people running Windows.

[Krasimir Angelov]

On 6 July 2018 at 10:48, John J. Camilleri <jo...@johnjcamilleri.com> wrote:

Yes, that is a very good point. There will inevitably be some logic which probably can't be handled by a Makefile alone, and there Bash scripts seem like the obvious solution. But I'm not sure what implications this has for people running Windows.

That was actually why I made the Setup.hs to build the RGL. In this way the Windows users don't need anything else than GHC which they would need anyway in order to build the compiler. Before switching to using cabal, the only way to build GF on Windows was via Cygwin which is a very unnatural environment on Windows. 

Splitting the RGL and opens the question again. There is no Make by default on Windows, so maybe, after all, using a build script in Haskell would be the most platform independent solution which doesn't add new dependencies to the eco system.

[John J. Camilleri]

I think the idea of being able to avoid Haskell completely is very enticing, especially for those who don't have Haskell installed but want to develop and build RGs. Maybe we can have a "light" build script for the RGL, in both Bash format and a Windows batch file, which does the bare minimum. Then for more complex build options we can use Makefile + Haskell.

Yes it's more stuff to maintain (and I have no experience with batch files nor a Windows machine to test them on), but if they are truly "light" then they should be very simple.

[Krasimir Angelov]

Another option is to add enough smartness in the compiler so that building the RGL doesn't need much more sophistication.

[John J. Camilleri]

I am currently putting together a new build script for the RGL in Haskell, essentially based on Setup.hs. I am assuming that lib/src/Make.hs is mostly outdated — please correct me if I'm wrong! Either way I am still looking closely at it in case there's something useful there.

I still think a Haskell-free way of building the RGL is important to have, but that will be a second priority after I sort out all the details of the primary build method.

John

[Aarne Ranta]

Right: lib/src/Make.hs was used for internal development purposes many years ago but has been replaced by the cabal procedure long time ago.

  Aarne.

On Tue, Jul 10, 2018 at 10:44 AM, John J. Camilleri <jo...@johnjcamilleri.com> wrote:

I am currently putting together a new build script for the RGL in Haskell, essentially based on Setup.hs. I am assuming that lib/src/Make.hs is mostly outdated — please correct me if I'm wrong! Either way I am still looking closely at it in case there's something useful there.

I still think a Haskell-free way of building the RGL is important to have, but that will be a second priority after I sort out all the details of the primary build method.

John

On Fri, 6 Jul 2018 at 13:10, Krasimir Angelov <kr.an...@gmail.com> wrote:

Another option is to add enough smartness in the compiler so that building the RGL doesn't need much more sophistication.

На пт, 6.07.2018 г., 11:27 John J. Camilleri <jo...@johnjcamilleri.com> написа:

I think the idea of being able to avoid Haskell completely is very enticing, especially for those who don't have Haskell installed but want to develop and build RGs. Maybe we can have a "light" build script for the RGL, in both Bash format and a Windows batch file, which does the bare minimum. Then for more complex build options we can use Makefile + Haskell.

Yes it's more stuff to maintain (and I have no experience with batch files nor a Windows machine to test them on), but if they are truly "light" then they should be very simple.

On Fri, 6 Jul 2018 at 11:16, Krasimir Angelov <kr.an...@gmail.com> wrote:

On 6 July 2018 at 10:48, John J. Camilleri <jo...@johnjcamilleri.com> wrote:

Yes, that is a very good point. There will inevitably be some logic which probably can't be handled by a Makefile alone, and there Bash scripts seem like the obvious solution. But I'm not sure what implications this has for people running Windows.

That was actually why I made the Setup.hs to build the RGL. In this way the Windows users don't need anything else than GHC which they would need anyway in order to build the compiler. Before switching to using cabal, the only way to build GF on Windows was via Cygwin which is a very unnatural environment on Windows. 

Splitting the RGL and opens the question again. There is no Make by default on Windows, so maybe, after all, using a build script in Haskell would be the most platform independent solution which doesn't add new dependencies to the eco system.

--

---
You received this message because you are subscribed to the Google Groups "Grammatical Framework" group.

To unsubscribe from this group and stop receiving emails from it, send an email to gf-dev+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

---
You received this message because you are subscribed to the Google Groups "Grammatical Framework" group.

To unsubscribe from this group and stop receiving emails from it, send an email to gf-dev+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

---
You received this message because you are subscribed to the Google Groups "Grammatical Framework" group.

To unsubscribe from this group and stop receiving emails from it, send an email to gf-dev+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

---
You received this message because you are subscribed to the Google Groups "Grammatical Framework" group.

To unsubscribe from this group and stop receiving emails from it, send an email to gf-dev+unsubscribe@googlegroups.com.

[John J. Camilleri]

The new split repositories are now available as BETA at the following locations:

https://github.com/GrammaticalFramework/gf-core

https://github.com/GrammaticalFramework/gf-rgl

These repositories have been created from the GF repository using this script.

I would appreciate it if some keen beta testers would clone the repositories above, build/install GF and the RGL using the instructions below, and let me know how it goes.

• GF is still installed with Cabal, i.e. just run cabal install from the gf-core repository.
  
• The one-liner for installing the RGL is make install — there are a lot more options which you can learn about in the README.
  

I imagine there will be some feedback and improvements to be made. Once we are happy with the new build scripts, the steps will be as follows:

1. I will announce a date when the official split will happen, maybe a week or so in advance.
2. On this date:
1. The GF repository will be archived, meaning it will accept no more pushes or pull requests (but you can still pull/clone).
2. The new repositories will be re-created using the script above
3. An announcement will be made, and from this point on all developers should clone the new repositories and start pushing/pull-requesting there.
   

Notes:

1. The GF repository will continue to exist (as an archive), meaning that no one's workflow should suddenly get broken if they don't start using the new repositories (although they will miss any future updates).
2. Pull requests and issues will not follow into the new repositories. It would be nice if at least some of these could be resolved before the split, but I will take care of copying over the ones I think are still valid as new issues in the new repositories.
3. There is a bunch of other things which will need to be tended to soon after the split, such as the package-building scripts, setting up continuous integration, updates to all the docs, etc. I'll start by maintaining a checklist.

John

On Wed, 11 Jul 2018 at 09:01, Aarne Ranta <aa...@chalmers.se> wrote:

Right: lib/src/Make.hs was used for internal development purposes many years ago but has been replaced by the cabal procedure long time ago.

  Aarne.

On Tue, Jul 10, 2018 at 10:44 AM, John J. Camilleri <jo...@johnjcamilleri.com> wrote:

I am currently putting together a new build script for the RGL in Haskell, essentially based on Setup.hs. I am assuming that lib/src/Make.hs is mostly outdated — please correct me if I'm wrong! Either way I am still looking closely at it in case there's something useful there.

I still think a Haskell-free way of building the RGL is important to have, but that will be a second priority after I sort out all the details of the primary build method.

John

On Fri, 6 Jul 2018 at 13:10, Krasimir Angelov <kr.an...@gmail.com> wrote:

Another option is to add enough smartness in the compiler so that building the RGL doesn't need much more sophistication.

На пт, 6.07.2018 г., 11:27 John J. Camilleri <jo...@johnjcamilleri.com> написа:

I think the idea of being able to avoid Haskell completely is very enticing, especially for those who don't have Haskell installed but want to develop and build RGs. Maybe we can have a "light" build script for the RGL, in both Bash format and a Windows batch file, which does the bare minimum. Then for more complex build options we can use Makefile + Haskell.

Yes it's more stuff to maintain (and I have no experience with batch files nor a Windows machine to test them on), but if they are truly "light" then they should be very simple.

On Fri, 6 Jul 2018 at 11:16, Krasimir Angelov <kr.an...@gmail.com> wrote:

On 6 July 2018 at 10:48, John J. Camilleri <jo...@johnjcamilleri.com> wrote:

Yes, that is a very good point. There will inevitably be some logic which probably can't be handled by a Makefile alone, and there Bash scripts seem like the obvious solution. But I'm not sure what implications this has for people running Windows.

That was actually why I made the Setup.hs to build the RGL. In this way the Windows users don't need anything else than GHC which they would need anyway in order to build the compiler. Before switching to using cabal, the only way to build GF on Windows was via Cygwin which is a very unnatural environment on Windows. 

Splitting the RGL and opens the question again. There is no Make by default on Windows, so maybe, after all, using a build script in Haskell would be the most platform independent solution which doesn't add new dependencies to the eco system.

--

---
You received this message because you are subscribed to the Google Groups "Grammatical Framework" group.

To unsubscribe from this group and stop receiving emails from it, send an email to gf-dev+un...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

---
You received this message because you are subscribed to the Google Groups "Grammatical Framework" group.

To unsubscribe from this group and stop receiving emails from it, send an email to gf-dev+un...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

---
You received this message because you are subscribed to the Google Groups "Grammatical Framework" group.

To unsubscribe from this group and stop receiving emails from it, send an email to gf-dev+un...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

---
You received this message because you are subscribed to the Google Groups "Grammatical Framework" group.

To unsubscribe from this group and stop receiving emails from it, send an email to gf-dev+un...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

---
You received this message because you are subscribed to the Google Groups "Grammatical Framework" group.

To unsubscribe from this group and stop receiving emails from it, send an email to gf-dev+un...@googlegroups.com.

[Aarne Ranta]

Hello John,

Both repositories install on my Mac just as you describe. Thanks for making it so clear and easy! 

Also thanks Bruno, Krasimir, and others for contributing to the discussions.

Regards

  Aarne.

To unsubscribe from this group and stop receiving emails from it, send an email to gf-dev+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

---
You received this message because you are subscribed to the Google Groups "Grammatical Framework" group.

To unsubscribe from this group and stop receiving emails from it, send an email to gf-dev+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

---
You received this message because you are subscribed to the Google Groups "Grammatical Framework" group.

To unsubscribe from this group and stop receiving emails from it, send an email to gf-dev+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

---
You received this message because you are subscribed to the Google Groups "Grammatical Framework" group.

To unsubscribe from this group and stop receiving emails from it, send an email to gf-dev+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

---
You received this message because you are subscribed to the Google Groups "Grammatical Framework" group.

To unsubscribe from this group and stop receiving emails from it, send an email to gf-dev+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

---
You received this message because you are subscribed to the Google Groups "Grammatical Framework" group.

To unsubscribe from this group and stop receiving emails from it, send an email to gf-dev+unsubscribe@googlegroups.com.

[bruno cuconato]

hello John,

I just cloned the repos (~150MB in all, so much faster!), and installed everything smoothly on my ubuntu 18.04 machine with GHC 8.0.2.

I'm excited to test the new RGL build script for building only one language, which should be much faster.

thank you for the great work!

-- bruno cuconato

To unsubscribe from this group and stop receiving emails from it, send an email to gf-dev+un...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

---
You received this message because you are subscribed to the Google Groups "Grammatical Framework" group.

To unsubscribe from this group and stop receiving emails from it, send an email to gf-dev+un...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

---
You received this message because you are subscribed to the Google Groups "Grammatical Framework" group.

To unsubscribe from this group and stop receiving emails from it, send an email to gf-dev+un...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

---
You received this message because you are subscribed to the Google Groups "Grammatical Framework" group.

To unsubscribe from this group and stop receiving emails from it, send an email to gf-dev+un...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

---
You received this message because you are subscribed to the Google Groups "Grammatical Framework" group.

To unsubscribe from this group and stop receiving emails from it, send an email to gf-dev+un...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

---
You received this message because you are subscribed to the Google Groups "Grammatical Framework" group.

To unsubscribe from this group and stop receiving emails from it, send an email to gf-dev+un...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

---
You received this message because you are subscribed to the Google Groups "Grammatical Framework" group.

To unsubscribe from this group and stop receiving emails from it, send an email to gf-dev+un...@googlegroups.com.

[John J. Camilleri]

Thanks for the positive first indications! I now want to fix the date for the split:

On Wednesday 25th July 2018 at 23:00 (GMT+2) the main GF repository will be archived, meaning that it will no longer accept any pushes or pull requests.

Shortly afterwards, the gf-core and gf-rgl repositories will be updated and become the GF project's new main repositories. An announcement will be made on this mailing list (in a separate thread) and the install instructions on the GF website will be updated.

To all developers: you should keep committing to the GF repository until the date above. After that, you will need to clone the new repositories and move your work there. Please prepare for this accordingly.

If anyone has any issues with the above, then this is the time to speak up!

John