rebar3 thoughts

[Bunny Lushington]

Hi All —

I’m wondering if the mailing list is a good venue for discussing Nitrogen’s potential migration to a rebar3/OTP configuration? I have skin in the game and some thoughts about what a rebar3 powered Nitrogen might look like but am interested in hearing what others needs and expectations are.

I know it’s possible with the current nitrogen_core codebase but how I’ve gone about it ignores features that are probably required for general use. (Specifically I’ve concentrated on a Cowboy based deployment and have eschewed the plugin, element, and page templating mechanisms.) While this is sufficient for me, it’s hardly reasonable to lose functionality as we move forward.

I have a few questions:

1. Much of the current “overlay” methodology could be replaced by the rebar3 template mechanism. Employing templates, it would be possible to execute `rebar3 new nitrogen-webmachine myapp` after installing the nitrogen-webmachine (or nitrogen-cowboy, &c.) template. This would create a familiar rebar3 app directory structure with requisite dependencies written into the rebar.config as well as pre-populating priv/static/{js,css,…} directories. The benefit of this is (after installing the template) it’s just the one command to create a new, working application. Is this a reasonable approach? Is there a better one?

2. It seems like the bin/nitrogen functions can be replaced with rebar3 directly. bin/dev functionality can, I believe, be replaced with a rebar3 plugin allowing for commands like `rebar3 nitrogen element new_element`. The alternative is to retain useful bin/* commands. I see benefits to both. For plugins: it’s “just” rebar3, anyone familiar with the tool will use it like they always have; plugins can be versioned and upgraded separately from the original installation. Keeping the existing bin/nitrogen retains familiarity for existing Nitrogen developers. Are there any thoughts around this?

3. Although maintaining the “legacy” artifacts (which are mostly related to directory structure) seems like an open question (specifically: is it required?) it appears that there would not be occasion to break compatibility with regards to nitrogen_core, simple_bridge, and so on. In other words, a “legacy” Nitrogen application and a rebar3 Nitrogen application would use the same essential libraries/applications. I assume this is preferable (it certainly is for me) but knowing how important it is generally might influence development.

4. How common is Docker in your environment? One of the gotchas about using templates is that the template needs to be installed before creating the application. This isn’t difficult per se but presumably minimizing the work involved bootstrapping a new application is important.

Please note: I don’t want to step on Jesse’s or anyone else’s work. Quarantine being what it is, I have more time than usual on my hands and would like to help move this line of work forward, but not in a silo. If anyone has thoughts or has been working on it and can use a contributor, please let me know!

Hope y’all are keeping healthy and (reasonably — we’re Erlang devs after all —) sane.

b.

[Lloyd R. Prentice]

Hi Bunny,

I don’t have the technical chops to respond specifically to your questions. But I do have strong opinions of a broader nature.

As an author/indie-publisher with an advanced degree in public affairs communication, I see Nitrogen as a powerful tool for human-to-human communication. To fulfill this potential several things must be emphasized:

1. Easy to learn and use — even by non-professional programmers
2. Fluency in all modes of human communication — text, images, audio, video
3. Easy to interface with or embed into other open source software
4. Easy to interface with networked hardware and sensors
5. Easy to coevolve with technology advances
6. Easy to deploy with high security

Clearly Nitrogen 2.4.0 falls far short of this idealistic vision. But it’s far closer than any other web-tech stacks I’ve looked into. An actively involved community is the key to closing the gaps.

Documentation and tutorials are the essential foundations of item 1. Jesse and I wrote and published Build It with Nitrogen with this foremost in mind. One of our hopes for the book is to recruit more folks into the Nitrogen community including students, entrepreneurs, and young programmers just kicking off their careers. But much more documentation work remains to be done.

Jesse and I have been kicking around the idea of a series of shorter project-oriented books that could possibly reach across the remaining items on my list above. But time and energy are serious constraints so we would much appreciate ideas and involvement from the Nitrogen community. What well-documented amazingly cool demo projects would you like to build or see built, published, and promoted?

One comment with regard to templates: I spent time earlier this year playing with markdown and Pandoc as a route toward rapid development of templates and fast prototyping of Nitrogen sites. Jesse has created an element in Ver. 2.4.0 to interface with Pandoc which, I believe opens up wide areas of application related to item 2 above. Happy to discuss if anyone is interested.

All the best,

Lloyd

Sent from my iPad

> On Dec 6, 2020, at 6:06 PM, Bunny Lushington <bu...@bapi.us> wrote:
>
> Hi All —

> --
> You received this message because you are subscribed to the Google Groups "Nitrogen Project / The Nitrogen Web Framework for Erlang" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to nitrogenweb...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/nitrogenweb/052CEF2B-3499-43DA-B18F-2CFB7C39FD04%40bapi.us.

[Bunny Lushington]

> On Dec 6, 2020, at 18:11, Lloyd R. Prentice <ll...@writersglen.com> wrote:
>
>
> As an author/indie-publisher with an advanced degree in public affairs communication, I see Nitrogen as a powerful tool for human-to-human communication. To fulfill this potential several things must be emphasized:
>
> 1. Easy to learn and use — even by non-professional programmers
> 2. Fluency in all modes of human communication — text, images, audio, video
> 3. Easy to interface with or embed into other open source software
> 4. Easy to interface with networked hardware and sensors
> 5. Easy to coevolve with technology advances
> 6. Easy to deploy with high security
>
> Clearly Nitrogen 2.4.0 falls far short of this idealistic vision. But it’s far closer than any other web-tech stacks I’ve looked into. An actively involved community is the key to closing the gaps.

[tl;dr — I need a life.]

Thank-you Lloyd for making explicit these essential goals!

I agree with your assessment that Nitrogen might fall short but it's
further along than analogous frameworks. Interestingly enough, I find
that a lot of potential anti-features actually work to the user's
advantage; to wit:

-- The lack of an ORM (loosely defined, of course there are no
objects per se). How one interacts with datastores is left up to
the user. For example, it's entirely possible to sequester all
SQL code from Erlang code. In terms of optimization and
reusability, this is huge.

-- No magic. The closest Nitrogen comes to doing something without
explicitly being told to is routing requests to modules. Reading
the code will give a fair understanding of what's going on
without understanding a baroque configuration.

-- Explicitly named functions. Okay, so this is an Erlang
(anti-)feature but still, it's trivial to figure out what module
a function is being called from by looking at the function call.

-- Freedom from MVC. While abstraction is sometime great, so it
immediacy. I'm reminded of Perl's guiding principle (at least it
used to be) "Make the easy things easy and the hard things
possible." You can implement MVC but for easy tasks, you don't
have to.

-- Agnosticism. Use whatever libraries you want, how you want. I'm
a firm believer in using UUIDv4 for surrogate keys in DBs.
You're not? Fine. I'm a Bootstrap fan. You're not? Fine
again. Being shoehorned into someone else's idea of what's the
Right Way too often leads to copious amounts of code (or worse:
configuration) trying to work around the mismatch.

So about this rebar3 stuff...

I think it's quickly becoming a standard way of developing, managing,
and deploying Erlang applications. It's pretty well known in the
community and conforming to its conventions will make life easier for
new Nitrogen developers. There's a balance with this: it's another
thing to learn if you don't know it (seemingly at odds with your point
#1) but it absolutely aligns with your goal #3. Given the
availability of rebar3 documentation and assistance available, I think
the pros outweigh the cons.

If we were to undertake this work, I foresee a few benefits, at least
for certain user cohorts:

-- (Easier) Elixir integration. There are two benefits: all of the
Elixir based libraries become available if we keep an eye towards
easy integration; and our user base potentially expands to not
only current Elixir devs but also non-BEAM developers who might
shy away from Erlang because of its unfamiliarity. (I personally
would benefit from integrating with the Elixir based Nerves
project, see your points #3 and #4.)

-- Container deployment. OTP applications are almost trivially
deployed on Docker. As it stands, if you want to do it with the
existing Nitrogen, you're kinda on your own (and there are tricky
bits). In the enterprise there's a big difference between
"Erlang application" and "Docker container" when you're asking
your systems group to manage it. (At least in my experience.)

-- Did I mention familiarity?

All of this is to say, I firmly agree with your notions of emphasizing
the human and appreciate the reminder that technology and incremental
changes should be focused on achieving your stated goals.

> Jesse and I have been kicking around the idea of a series of shorter project-oriented books that could possibly reach across the remaining items on my list above. But time and energy are serious constraints so we would much appreciate ideas and involvement from the Nitrogen community. What well-documented amazingly cool demo projects would you like to build or see built, published, and promoted?

There are a number of demos/plugins/applications/techniques (take your
pick) that I think would be beneficial:

-- Database access! What does it look like to interface with an
RDBMS? How is SQL managed? How are schemas created and
maintained? What's an appropriate level of abstraction?

-- DTL (Django Template Language) integration. How can page design
be separated from back-end processing? How do DTL templates
interact with #template{}? Is it possible to specify page
interactions from the DTL (or template HTML) side instead of
relying on Erlang code?

-- User and group management; login/logout; session management. How
many times have you sat down to write an application and first
had to either hack out user/group CRUD before you could get to
the business problem? Would a selection of plugins to manage
these common tasks be helpful? How about an AD/LDAP interface?

-- API management. Even if there's a page I can visit to, e.g., add
a user, how can I leverage that code as part of an API? How can
such a system be self-documenting and discoverable?

I'm 100% sure this list could go on. It should also not be taken as a
shortcoming of the framework that these functionalities are not baked
in. (See "Agnosticism" above.) But providing some starting points
would be wicked user friendly.

I apologize for the length of this email; you hit me where I live. I
firmly believe that expressive languages and libraries do more to
advance practical goals than any amount of fancy bit twiddling (when
fancy bit twiddling isn't actually called for). I don't want to knock
out a prototype and then re-write it to make it production-ready; I
want to write production code from line one.

b.

[Allan Streib]

Anyone have an opinion on erlang.mk vs rebar3?

I don't have enough experience to have strong opinions, but having worked in Unix environments for nearly 3 decades I find Makefiles to be pretty natural. I've done a couple of small projects with erlang.mk and it worked well for me.

https://erlang.mk/guide/why.html

I know rebar3 is the "official" build tool with all the momentum so maybe there's not a good reason to use anything else.

[Anton Prokofiev]

Hello, Bunny

As for myself, I am using nitrogen with Cowboy web server, so other servers are not actual for me.

Related to rebar3 migration:

I use erlang.mk a as a building tool that already isolate me from rebar.

I would recommend to take a look on it as release tool:  this probably will simplify integration of nitrogen plugins.

Regards, Anton

[Jesse Gumm]

Whew, there's a lot here.  Thanks for all the feedback.

A few things, and only briefly:

1) The plan is to make the conversion to rebar3, and indeed support rebar3 templates for deployment.  That said, the "anointed" methods won't be going away.  Running `make slim_yaws` would still generate a yaws release, with the proper rebar3 configuration and directory structure.  But it will use symlinks to continue support for the legacy structure. To go along with this, I'm working on a "convert-to-rebar3" script so if you're using the standard generated directory structure, that new script will properly convert it to a rebar3-ready project.

2) bin/nitrogen is ultimately an older release-generated then slightly modified rebar start script.  With rebar3, you're right, it's redundant.  I will likely end up sticking a skeleton script in its stead that would end up calling the appropriate rebar3 start script, though again with some mild hackery, because, frankly, I like that Nitrogen supports having multiple .config files in etc/ that get loaded during startup, rather than having to require all your config to be in one app.config file.  That said, the 'bin/dev' script I never actually use myself, but I agree that a rebar3 plugin or template would likely be the simplest way to emulate that functionality.

3) Database Access: I've specifically shied away from anointing one true database integration technique.  In our book, we cover connecting to DETS, mnesia, and postgres (though in my daily life, I actually use mysql).  The RDBMS I use on a daily basis goes through https://github.com/choptastic/sql_bridge (which can connect to mysql or postgres - and I plan on adding sqlite at some point).

4) DTL: I'm definitely open to adding DTL support, though I'm torn between using the existing #template element or creating a new #erlydtl template.  I suspect the latter is probably more prudent. but I'm not a regular DTL user.  I've written support for it in the past (back in my ChicagoBoss days, particularly), but not much beyond that.

5) rebar3 vs erlang.mk.  My plan is to initially support rebar3. Once the dust has settled on that, I'm not opposed to adding erlang.mk support.

The short of all this is that the next *major* thing I'm working on is the rebar3 rework and all that that entails.  The goal is to make it "standard" rebar3 setup, while ensuring backwards compatibility with existing Nitrogen applications.

Again, I appreciate all your feedback and suggestions here.

Also, as a reminder, the official roadmap (if you can call it that) is up at trello.nitrogenproject.com.

-Jesse

--

You received this message because you are subscribed to the Google Groups "Nitrogen Project / The Nitrogen Web Framework for Erlang" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nitrogenweb...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/nitrogenweb/eaec92e3-5d36-4dd0-ae4e-2ec642a7deb0n%40googlegroups.com.

--

Jesse Gumm
Owner, Sigma Star Systems
414.940.4866 || sigma-star.com || @jessegumm