General ways and concrete examples of approaching web dev

[Marc Kaufmann]

Hi all,

TL;DR: What are some patterns/approaches for web sites for which Racket is particularly suited *and* for which you can point at decently written up examples in Racket or other languages that share such features?

Much longer me:

I spent much of summer implementing online web sites with quite a bit of state (do step 1; if answer a do step 2, if answer b do step 3; keep forking, randomize some things in between). The first website I did in Django, because 2 years ago when I rolled my own in Racket, I ended up with an unmaintainable mess of code that had sql and sexpr sprinkled across all types of files, and I didn't dare touch anything anymore. Now I learned that I am perfectly capable of writing an unmaintainable mess of code in Python too. Hence the second such web site (all are used for economic experiments, think surveys with a lot of randomization and if/then logic) is again back to Racket.

The thing is that, while I know about MVC and grok it better after having spent time with Django, I am often at a loss to figure out where to draw boundaries in Racket. This is true for Django too, but there are so many examples out there, that it is easier to figure out what belongs where, and Django forces you to put some things in the models (more or less) and others in the views. And all the standard stuff - forms, simple SQL - has obvious ways of doing it. There are some things I dislike about it, but these are some positives.

Now that I am back in Racket land, and at least partially get what continuations do (thanks to primarily Philip, George, Jesse, Bogdan and Matthew B's writings and answers on this list over the years), my code is vastly better than 2 years ago. Yet, I still feel that it is pretty bad, and I see loads of duplications that I do, sometimes due to lack of time to refactor, but just as often because I have no clue how to refactor it in a way that will not make it ridiculously fine-tuned to my current purpose.

So, my question is this: what approches (if you like big ponderous words, what paradigms) and patterns are there to write web sites, and are you aware of examples that highlight them *concretely* in code in a way that the send/suspend/dispatch is described in the "Continue: ..." tutorial? Here are approaches that I know of:

- MVC: Django and Rails for instance, easy to find examples. Usually object oriented it seems, but it's sort of what I do in Racket as well.

- Single-Page Apps: I never implemented one, but my sense was that you can do MVC with it too, and it's more about whether you use JS to do things asynchronously. I may be wrong and there might be more to it.

- Continuation style: What "Continue" does, and again it's not really an alternative to MVC, but more how you implement it (or other things). I still struggle a lot with doing this well, as there are few descriptions of it that go beyond "Continue". When I google, it brings up essentially the Racket crowd and something called Seaside (I had never heard of it)

- Microservices (I do not get what it is, if it even is one thing)

So what are examples in Racket or Racket-like languages (semantics, not syntax, so it doesn't have to look Lispy if it is functional or uses continuations or whatever features of Racket) that I could use to guide how I *could* structure my web site in a given way? Django isn't that great a fit since it has a pretty heavy ORM, and does lots of magic behind the scenes on objects, which is not trivial to map to the functional style of Racket.

I don't know whether these are related to it, but half (all?) of my struggles come from trying to manage the state people are in and ending up passing around biggish objects, or adding 2 or more boolean fields per page to the database and calling the database on every new page.

I understand that this is not the most well-formed question on this email list and it's likely I am asking the wrong question, so I am very open to being told "What you should be asking is ... " (such as, this is not about web stuff, but ...).

Cheers,
Marc

[Jens Axel Søgaard]

Hi Marc,

Den tir. 24. sep. 2019 kl. 21.28 skrev Marc Kaufmann <marc.ka...@gmail.com>:

TL;DR: What are some patterns/approaches for web sites for which Racket is particularly suited *and* for which you can point at decently written up examples in Racket or other languages that share such features?

A big question on which I will punt.
 

I am still a beginner in the art of crafting web sites, but here is my perspective.
There are many ways to organise web apps. Most patterns that work in other languages will also work in Racket.

A popular approach is to make the web app "stateless" meaning that all state is stored on a database.

This makes scaling easy, just add more web-servers and let the load balancer direct the traffic

to the web server with minimal load.

The MVC approach fits well here. 

The model takes care of representing and storing data in the database.

All "business logic" is handled in the model as well.

The view produces html pages based on data to be presented.

The view never calls functions provided by the model directly.

The control receives a request, determines which model operation to call,
get any additional data and passes the the results on to the view.

There are choices to be made at all levels. This is what makes it confusing to begin
developing web apps.

Should the model use plain lists/vectors, structs or objects when representing data? 

For the first choice  the package `db` will do. For structs `deta` is a nice choice and for objects `racquel`.

How is the view representing html pages?
Some of the choices: strings, s-exprs, x-expres or html-structures?
(by html-structures I mean: define a struct for each html tag)

Should the control use `continuations`? 

For single page apps you need basically write a javascript program that
generate requests and your control can them handle them in the same

way you would in a server side web app.

It's worth experimenting with different approaches to see what fits your style.
One of the goals of (soon to be released) racket-stories.com (previously known as web-tutorial/listit)
was to experiment with different tools and at the same time have a "production web-site"
in Racket with source available. What the site actually does is less important.

The source is available here and contains more comments than I usually write:  

    https://github.com/soegaard/web-tutorial

The listit1-listit3 are the initial versions, which might become parts in a tutorial at some point.
The almost-ready-to-release version is in version4.

The choices I made:
   - the model uses structs to represent data  (and `deta` to handle the struct to db work)
   - the view   uses  html-structs
   - the control doesn't use continuations (in order to make the web-app stateless)   

Next up on my list of things to experiment with is "single page apps". 

/Jens Axel

 

[Marc Kaufmann]

Thanks for the response to a rather general question. I'll definitely have a look at your code for Racket stories, which is live now if I see correctly. Nice!

I guess one concrete thing that I find surprisingly hard in my code is to get rid of the number of times I repeat the structure of the data:

1. write the SQL code to create the database

2. write the interface in Racket that deals with the database directly (transform into Racket values, but it's still in the same groups)

3. write functions that need to know about this structure, because they extract the information from part 2, but different fields for different uses

4. write the view that - quite often - looks like a subset of the database and in my case calls functions from part 3, which again repeats huge parts of what is in part 1

So if I have a user with 5 fields I am interested in, I find that if I change part of my database schema, I usually update 3 (or more) places, which tells me that I didn't get the abstractions right. (And I haven't added tests, or else I'm sure I'd have another layer or 2 of repetition.) The continuation passing style actually suits my use cases much better than storing state in the database -- except that I need to be able later on to recreate the path a user took, in order to figure out what happened, so I have to store it in an easy-to-interpret way in the database.

My guess is that I don't grok MVC, and therefore end up with all these repetitions, so I'll have a look at your web site -- although part of the reason I don't grok it may be that most beginner resources have websites that are less stateful than what I do, which is much more like a game with many action nodes and randomization. I should probably have a look at Realm of Racket and see if I can't port the big-bang ideas to the web server.

Thanks for the reply and the tutorial, I'll have a look.

Cheers,

Marc

[George Neuner]

Rearranged a bit for continuity ...

On Tue, 8 Oct 2019 19:44:55 +0200, Marc Kaufmann
<marc.ka...@gmail.com> wrote:

>Thanks for the response to a rather general question. I'll definitely have
>a look at your code for Racket stories, which is live now if I see
>correctly. Nice!
>
>I guess one concrete thing that I find surprisingly hard in my code is to
>get rid of the number of times I repeat the structure of the data:
>
>1. write the SQL code to create the database
>2. write the interface in Racket that deals with the database directly
>(transform into Racket values, but it's still in the same groups)
>3. write functions that need to know about this structure, because they
>extract the information from part 2, but different fields for different uses
>4. write the view that - quite often - looks like a subset of the database
>and in my case calls functions from part 3, which again repeats huge parts
>of what is in part 1
>
>So if I have a user with 5 fields I am interested in, I find that if I
>change part of my database schema, I usually update 3 (or more) places,
>which tells me that I didn't get the abstractions right. (And I haven't
>added tests, or else I'm sure I'd have another layer or 2 of repetition.)

> :

>
>My guess is that I don't grok MVC, and therefore end up with all these

>repetitions ...

It remains to be seen whether you really understand what MVC is about
... but repetition unfortunately is a major issue with it, as it is
with many other application design patterns.

In general, embracing MVC as a design philosophy won't reduce your
work at all - in fact it likely will increase it. MVC does nothing
whatsoever to reduce the application's unique processing - it only
addresses ways to structure the application. But usually it will
produce a program that is more modular and clearer to understand.
[But your observations about maintenance are quite valid.]


One of the issues I have with MVC is the way it typically is
presented: data flow in the tutorials tends to be exaggerated and
overly complicated, and the explanation typically revolves around DBMS
and graphical UIs that simply aren't relevant to it.

As a design pattern, MVC dates from the 1960s, from the era of CRT
display/edit form-based applications. The View and Controller
historically were integrated, and the "Model" was a memory cache of
one or more data records.
[Relational DBMS didn't exist in the 60's, but there were many
varieties of file based record systems. Dedicated DBMS software based
on network/graph and hierarchical models appeared in the 70's, and
then relational models appeared in the 80's.]

Obviously MVC can be - and has been - generalized to separate the
various concerns, divorce the "Model" from its "data record" origins,
and to allow more complex communication/control patterns. But the
modern notion of MVC has become so interwoven with DBMS, graphical
UIs, event handling, reactive programming and web services that people
learning about it often have a hard time figuring out what it really
means and how to use it effectively for their own programs.

MVC is simple when you understand what it was designed for originally.


Then there is the proliferation of simplistic MVC object "frameworks".
These typically address DBMS backing storage for the Model and little
or nothing else. Trying to use them for rapid application development
too often results in your application doing all the work, and your
high powered DBMS server being turned into little more than a very
expensive file server. <frown>


> :

>The continuation passing style actually suits my use cases much better
>than storing state in the database -- except that I need to be able
>later on to recreate the path a user took, in order to figure out what
>happened, so I have to store it in an easy-to-interpret way in the
>database.

> ... part of the reason I don't grok it may be that most

>beginner resources have websites that are less stateful than what I
>do, which is much more like a game with many action nodes and
>randomization. I should probably have a look at Realm of Racket and
>see if I can't port the big-bang ideas to the web server.

MVC per se has no problem with large amounts of state - the scope of
the Model, and how and where it is maintained, largely is irrelevant.
There doesn't need to be one big unified data store - there can be
many little ones.

I think part of your problem it is that you are integrating/conflating
what are purely implementation issues - such as whether to pass state
via continuations vs storing it in DBMS - with higher level design
issues like whether to use MVC vs some other equivalent architecture.




On the subject of database development, your best bet for reducing
effort is to invest in a database modeling tool.

There are many free and/or reasonably priced (for single developer)
tools that will generate DDL and simple select/update DML or view code
for you. The better ones also can generate morphing DDL for
structural alterations (though IMO that's better done manually), and
they offer other niceties like graphical views of your tables and
their relationships / linkages, easy manipulation of references (e.g.,
draw an arrow from one table to another to create an FKEY reference),
structural consistency checking, data display/edit, and other goodies.

High end [read "much more expensive"] UML/ERD tools can generate
application frame code from database structures and process
flowcharts. However, offhand I'm not aware of any that generate
Scheme or Lisp code - it almost always is Java or C++ (or more
recently C# or node.js Javascript).

Most likely, there's something you'll like that will fit your budget.


Hope this helps,
George

[Marc Kaufmann]

> I think part of your problem it is that you are integrating/conflating
> what are purely implementation issues - such as whether to pass state
> via continuations vs storing it in DBMS - with higher level design

> issues like whether to use MVC vs some other equivalent architecture.

Yep, I agree, and I actually know it - but that doesn't help me know what I shouldn't and what I should conflate. This thread has helped somewhat. If you know of any in-depth resource/book (whether modern web dev or old '80s MVC) that might help, I'd appreciate it.

> On the subject of database development, your best bet for reducing
> effort is to invest in a database modeling tool.

You don't happen to have a free/cheap (Linux compatible) one to try out? I found 10 lists with 'best 6/10/33 db modeling tools', which is 5/9/32 tools too many...

For now I'll have to roll my own due to deadlines, but I'll have a look. And see, I didn't even know such things existed (and am still not quite sure what part of the work they are for, but we'll see).

Cheers,

Marc

--
You received this message because you are subscribed to a topic in the Google Groups "Racket Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/racket-users/7cBrSU8aTS0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to racket-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/racket-users/u5jqpedeh2vhpf6tsfmn2333msr1hoicpv%404ax.com.

[Hendrik Boom]

On Wed, Oct 09, 2019 at 03:29:51AM -0400, George Neuner wrote:
...
...

> One of the issues I have with MVC is the way it typically is
> presented: data flow in the tutorials tends to be exaggerated and
> overly complicated, and the explanation typically revolves around DBMS
> and graphical UIs that simply aren't relevant to it.
>
> As a design pattern, MVC dates from the 1960s, from the era of CRT
> display/edit form-based applications. The View and Controller
> historically were integrated, and the "Model" was a memory cache of
> one or more data records.

That's what always bothered me about MVC -- the idea that the View and
Controller should be separate. The idea of interpreting user actions
(clicking and typing on a screen) without knowing details of the view
(which governed where he clicked and typed) seemed absurd.

Separating the model made sense. Especially because you might need very
different Views on different terminal devices.

-- hendrik

[George Neuner]

On Wed, 9 Oct 2019 10:01:34 +0200, Marc Kaufmann
<marc.ka...@gmail.com> wrote:

>You don't happen to have a free/cheap (Linux compatible) one to try out? I
>found 10 lists with 'best 6/10/33 db modeling tools', which is 5/9/32 tools
>too many...

<grin> I know what you mean.

I routinely deploy on Linux, but much of my development work actually
is done on Windows. Unfortunely, I'm much less familiar with the
state of DBMS tools on the Linux side.

There are some well respected programs that have Linux versions, but
many/most of them will require JVM and JDBC drivers for your DBMS ...
which may make their installation a bit tricky. I've seen plenty of
issues with JVM software on Linux - some programs just aren't happy
with OpenJVM/JDK and demand Sun/Oracle's version instead.


You didn't mention what DBMS you are targeting - that matters for
scripting - but I would take a look at the list below. They all are
free or have free versions, are multi-platform, and support multiple
DBMS.

https://www.oracle.com/database/technologies/appdev/sql-developer.html
https://dbeaver.io/
http://www.bestofbi.com/page/architect
http://squirrel-sql.sourceforge.net/
http://www.modelsphere.com/org/


George

[Marc Kaufmann]

Great, I'll have a look at these - and I'm mostly using postgres, although I don't have strong views.

--
You received this message because you are subscribed to a topic in the Google Groups "Racket Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/racket-users/7cBrSU8aTS0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to racket-users...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/racket-users/1bdtpehmfhajjoaasq97h5icachdkr6b9s%404ax.com.