Nitrogen 3.0 Progress Update

[Jesse Gumm]

Hey Folks!

For those of you wanting to know what's happening with Nitrogen 3.0, here are the latest developments. Some of these I've shared in the Nitrogen Slack Channel (#nitrogen at https://erlef.org/slack-invite/erlef) already, but for those of you that haven't here are the big updates.

But first, if you want to experiment with Nitrogen 3 functionality, there is a `rebar3` branch in nitrogen/nitrogen, nitrogen/nitrogen_core, and nitrogen/NitrogenProject.com.

Also the roadmap and what is being worked on is kept up-to-date at trello.nitrogenproject.com

Build Process

The biggest and most fundamental part of Nitrogen 3 is the upgrade to rebar3 as the official build tool (as you may have inferred from the branch name). The process right now is almost the same, except I'm putting more emphasis on using `make` instead of relying on non-standard modifications to rebar-generated scripts (as Nitrogen 2.0 originally did).

Creating a new project from the nitrogen directory now can be as simple as `make build` - this is a new terminal-based menu for creating a project. You can still do things like `make slim_yaws PROJECT=my_awesome_app` but `make build` eliminates the need to remember the arcana.

This rework has also simplified the deployment and upgrade processes. For example, starting a production-ready release is now:

`make run_release`

Then when changes have been done to the codebase or dependencies, you can run:

`make upgrade_release`

That upgrade_release will generate all the release upgrade (relup) files, do all the work of deploying and performing a hot-upgrade.

And it has all the benefits of using rebar3: following a standard Erlang application structure, getting relx built in (over reltool, which is far trickier to get right), and being able to rely on hex.pm for packages.

This build process has simplified building an environment on Windows as well, as a little perk.

Vessels

As for new functionality, "vessels" were added as a new attribute to go along with postbacks. This allow "localizing" postbacks, and the fields wrapped in them. Basically, you can say something like: "when performing this postback, only count the fields contained within this nitrogen element (say, a #panel element)." You can see the vessel functionality in the demo page written in NitrogenProject.com - https://github.com/nitrogen/NitrogenProject.com/blob/rebar3/src/demos/demos_vessel.erl

New Session Handler

Nitrogen 3 will now, by default, use a new session handler based on a new application called Canister (https://github.com/nitrogen/canister). Canister is an mnesia-based distributed session server that aims for "eventually consistent." All copies of data exist in all nodes in the cluster, and the session info being in mnesia ensures durability after a node or cluster restart. This represents a big improvement over the Nitrogen 2 default simple_session_handler, as session info didn't survive a node restart, and could lead to some pretty serious split-brain problems if data ended up on two different nodes in the cluster. You can see the new handler here: https://github.com/nitrogen/nitrogen_core/blob/rebar3/src/handlers/session/canister_session_handler.erl The original simple_session_handler will remain in the system available to use if you prefer that.

Websocket Overloading

Nitrogen 3 is introducing websocket overloading. Basically, this allows you to use the already-established websocket connection in Nitrogen to send and receive messages that aren't just postbacks or postback responses. Along with that is a new websocket handler, which further allows you to deviate from even this already modified websocket behavior.

You can see a sample of how this works in the new websocket demo: https://github.com/nitrogen/NitrogenProject.com/blob/rebar3/src/demos/demos_websocket.erl

This functionality can allow for easier implementing of high-message-count applications like games, since those messages won't have the same overhead as the usual postback or "comet" messages.

jQuery

Work has started in attempting to remove jQuery as a core dependency. It's not yet complete, and likely won't be finished for the full Nitrogen 3.0, but still the work has begun and will likely be a slow update.

jQuery Mobile has not yet been removed, but will be removed and split into its own plugin project shortly. So if you do actually still use jQuery mobile, it will still be possible to upgrade to Nitrogen 3. That said, no work is planned beyond that for jQuery Mobile.

Handlers

If you followed in slack, you'd know I was doing some handler-system experiements and reworks. My work there proved fruitless for any real performance improvements. The <1% improvement I saw did not justify the added complexity of the code, so that work has been largely scrapped.

That said, I have a few new handlers planned:

The validation system is slated to be gutted and reworked as a validation system, allowing stripping out liveValidation and replacing with your own custom validation system, if so desire.

A modal handler is planned for dynamically launching a modal window. And a a confirm/prompt handler is planned for making your own popups. This would allow reimplementing popups as either javascript "alert" or javascript "confirm" popups, doing something with bootstrap or other JS library, or implementing your own functionality. I'm undecided on if those will be two different handlers or not. Likely, my plan is to have the "confirm/prompt" handler rely on the "modal" handler for the default setup.

I'm also planning a "notification" handler, which just controls little corner-style popups.

Performance Improvements

While the handler experiment yielded no fruit, I did manage to find some performance improvements elsewhere in the rendering engine, so I consider it a win. For example, I managed to reduce the "reduction" count and total memory used for handling a request and rendering the /downloads page in NitrogenProject.com by about half.

Redundant Classes

As many of you would have noticed, Nitrogen 2.x (and maybe even going back to Nitrogen 1), there is a redundancy in the classes of elements. For example: #textbox{} would produce <input type=text class="textbox wfid_temp123123">. That whole "class=textbox" is largely redundant. By default, Nitrogen 3 will no longer have these for the basic elements (compound elements are different). If, however, you have a lot of CSS that relies on these elements, there will be a section in rebar.config that will enable that functionality for you. This should help with integration with CSS and JS libraries where there are terminology conflicts, such as "class=label" having a specific meaning.

Upgrading to Nitrogen 3 and Backwards Compatibility

While Nitrogen 3 represents a major version increase, as usual, I'm taking extra special care to ensure backwards compatibility. I want upgrading to N3 to be pretty straightforward.

For one, there is a new upgrade script that will take a standard Nitrogen 2-based application, do a little magic, and you should have a *mostly* ready-to-go N3 structure, complete with rebar3, and updated rebar.config, and directory structure updates. The content of this script can be found here: https://github.com/nitrogen/nitrogen/blob/rebar3/scripts/convert_to_rebar3.sh

Further, I've tested the Nitrogen 3 branch in a Nitrogen 2-generated project, and it passed all tests, which means you could continue to use rebar2 instead of rebar3, along with the old directory structure, with minimal problems. Not that I recommend it, but that's a demonstration of the commitment to backwards compatibility.

What's left before Nitrogen 3 is released?

As mentioned, I have a few other handlers I want to rework, and a little more work on removing jquery (again, it jquery will almost certainly still be in N3, but it's being phased out). I also plan to add few more elements, actions, and functions, and of course, a bunch of new documentation still has to be written (a lot has been done, but more still to go).

Also, while grid 960 was cool in 2008, the built-in grid system needs to be reworked to be responsive. I'm still a little on the fence on the direction to go with that.

Further, the Nitrogen homepage is getting a much needed redesign.

Release ETA?

While I've been saying "Nitrogen 3 should be ready in a few months," that remains the answer. I'm still working on Nitrogen 3 mostly in my after-hours time, and I'm rapidly entering my busy season for work.

So while I'd like to have the release ready within a month, more realistically, it may be end of summer or early fall.

But given that nitrogen_core for N3 works in a N2-generated application, I plan on merging the rebar3 branches into the master branches pretty soon. I just need to be careful with that because I don't want folks following the Nitrogen Book (builditwithnitrogen.com) or the online tutorials to be stuck because the master branch no longer works with older documentation.

Jesse Gumm http://jessegumm.com

[Lloyd R. prentice]

Hi Jesse,

Much appreciate the update. But I don’t have sufficient tech chops to understand how the updates make it easier to develop a site. Clearly you’re investing a lot of work. Can you provide a Cliff notes version that highlights key benefits  and added value in nontechnical language?

Best,

L.

Sent from my iPad

On Mar 27, 2023, at 11:25 AM, Jesse Gumm <sigm...@gmail.com> wrote:



--
You received this message because you are subscribed to the Google Groups "Nitrogen Project / The Nitrogen Web Framework for Erlang" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nitrogenweb...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/nitrogenweb/9FDA3C7D-F646-4B95-939B-F8EB4852AC33%40getmailspring.com.

[Jesse Gumm]

It's a little tough to try to summarize without being technical, but the broad strokes come down to a few important things.

The TL;DR is this:

1) Standard Erlang application directory structure

2) Standardized on the rebar3 build tool

3) Less finickiness in adding an app to source control (git)

4) Simpler and safer release management, specifically in performing hot upgrades.

The longer story is this:

Nitrogen 2 projects (that use the build scripts, like `make rel_yaws`) relied on a number of modifications to a generated rebar release and a non-standard directory structure. Technically, rebar was generating a release and calling it a "dev environment." But this environment used a non-standard directory structure to work around the fact that Nitrogen existed long before "slim releases" were a thing. This caused confusion about what directories needed to exist in source control (especially before slim releases were introduced[1]). Originally, the instructions were "just add 'site' and 'etc' to source control", but that was a hassle - to deploy, you then needed to know that you'd later copy 'site' and 'etc' into a newly generated project (again, that's also a pain).

After slim releases were introduced, you could build a slim release and add almost the whole directory structure: mostly just leaving out the 'releases' and 'lib' directories.

* 'lib' was meant to be skipped because those were a bunch of dependencies that would be pulled by rebar, and if you didn't skip it, rebar would freak out.

* 'releases' was meant to be skipped because if you copied your project to another machine with a different version of Erlang, the build process would fail because it's looking for a different Erlang version as the basis for the slim release.

This lead to the `make fix-slim-release` script that would take a nuclear approach to the 'releases' directory, replacing it with a new one built for the version of erlang on the local machine.

Those historical problems have been largely alleviated by rebar3. Basically the entire Erlang ecosystem has migrated to rebar3 - Nitrogen is one of the last to complete that migration. In fact, I've been maintaining my own fork of rebar2[2] because

1) Nitrogen still relies on rebar2,

2) rebar2 mainline has been in read-only mode for 5 years, and

3) changes in Erlang have required updates to rebar to ensure it continues to work with Erlang 23+.

As a result, the astute user might notice that cloning nitrogen/nitrogen now actually builds a custom copy of rebar2 from my fork.

Now, Nitrogen3 and rebar3, deployment to a new machine is simpler: you can add the whole project to git, clone it on another machine (or do it with a tarball if you want, but I prefer git), then run:

`make run_release`

This will pull dependencies, build the app for your project, and run the generated release on your target machine in the console.

Alternatively, you can start it in daemon mode with:

`make daemon` (or `make start` - they're aliases of eachother in the makefile)

What makes this different than Nitrogen 2.0 is the general simplicity of hot upgrading the release and dependencies.

In Nitrogen 2.0, upgrading dependencies was quite a hack and not a durable if you wanted to do to a live machine. It was pausing sync with `sync:pause()`, then pulling the dependencies individually (or doing a `make update` to update all dependencies), pulling the newest version of your codebase, then running `sync:go()` and praying that everything would recompile properly. But it wasn't atomic, and if something failed, sync just continues to chug along recompiling modules, and you could end up with your production system in an inconsistent state. At this point, the solution usually required recompiling the whole thing with `make` then restarting the VM completely.

With rebar3 and some of the helper scripts in Nitrogen 3, you would still update your dependencies however you like `./rebar3 upgrade package_name` for example to get the latest version of `package_name` but when it's time to actually go live with the updates, you can run `make upgrade_running`.

That will generate a new release, create the appup scripts, then execute an actual release upgrade. If part of the release upgrade fails, the system will not be updated at all, and you'll be safe from an inconsistent system state.

On top of that, dependencies in rebar3 (at least those in hex) are easier to specify as well:

nitrogen_core

rather than

{nitrogen_core, {git, "https://github.com/nitrogen/nitrogen_core", {branch, master}}}

I know that's a lot, but hopefully that makes some sense.

-Jesse

[1] You can see the adventure I went on to try to grok slim releases in a blog post I wrote about 10 years ago: http://sigma-star.com/blog/post/nitrogen-slim-release

[2] https://github.com/choptastic/rebar

Jesse Gumm http://jessegumm.com

To view this discussion on the web visit https://groups.google.com/d/msgid/nitrogenweb/1D6C2667-EA4F-4273-8438-BF4AB418A52F%40writersglen.com.

[Lloyd R. prentice]

Excellent! 

Many thanks, Jesse, for both this clear update and your outstanding work on Nitrogen.

Best,

L.

Sent from my iPad

On Mar 28, 2023, at 12:38 PM, Jesse Gumm <sigm...@gmail.com> wrote:



To view this discussion on the web visit https://groups.google.com/d/msgid/nitrogenweb/F5A1DB4A-DB97-4D3C-B48E-25022ED818A6%40getmailspring.com.

[Carlo Ascani]

Il giorno lunedì 27 marzo 2023 alle 20:25:51 UTC+2 gu...@sigma-star.com ha scritto:

Also, while grid 960 was cool in 2008, the built-in grid system needs to be reworked to be responsive. I'm still a little on the fence on the direction to go with that.

There are tons of "css grid frameworks" out there that supports basic responsivness, (they go 1 column on small viewports...). That could be an idea.

Example: https://simplegrid.io/

 

Further, the Nitrogen homepage is getting a much needed redesign.

I could help with that.

 

While I've been saying "Nitrogen 3 should be ready in a few months," that remains the answer. I'm still working on Nitrogen 3 mostly in my after-hours time, and I'm rapidly entering my busy season for work.

So while I'd like to have the release ready within a month, more realistically, it may be end of summer or early fall.

Kudos. And thanks a lot.

I am playing with Nitrogen on my spare time, following the book.

 

But given that nitrogen_core for N3 works in a N2-generated application, I plan on merging the rebar3 branches into the master branches pretty soon. I just need to be careful with that because I don't want folks following the Nitrogen Book (builditwithnitrogen.com) or the online tutorials to be stuck because the master branch no longer works with older documentation.

Thanks a lot, I firmly believe that the hands on approach of the book really help here, as if you encounter problems you would likely ask for help or go to Github, where there will be a clear message about the new rebar3 feature :)

Kind Regards,

Carlo