current status of Curv

[Doug Moen]

I currently have 4 active projects in various states of completion, to improve Curv.

1. WASM executable. The goal is to make available some currently existing APIs for running Curv programs, so that Curv can run inside a web browser. Nothing needs to be redesigned at this point (phase 1).

2. The noise & pattern libraries. The goal is provide access to some well designed and easy to use APIs for creating fractal and cellular noise, to use it to create procedural textures and geometry.

3. Language improvements. Fixing various language limitations that get in my way, and simplifying the language (removing features no one will ever use).

4. New shape compiler. This is infrastructure work that's needed to remove limitations on what Curv programs can be successfully compiled into GLSL shader programs. It will allow some powerful new language APIs to be developed.

I also have a bunch of ideas on how to improve the underlying graphics engine: image import, voxel support, mesh support (eg, STL import), physically based rendering (PBR). but I'm not working on that any more until next year. It's waiting on the new shape compiler, and the new WebGPU standard. WebGPU will provide a modern, high performance, cross-platform GPU interface that is available on every platform, including the web, with C/C++/Rust implementations available for native applications on Linux, MacOS, Windows. There are preview implementations of WebGPU available, but it isn't ready for general use. This standard is being designed by the web browser manufacturers, who have decided that Vulkan is too unsafe, and too difficult to use, to be made into a web API.

The noise library will contain high quality implementations of various low level mathematical algorithms for generating noise. The pattern library will contain a high level, composable interface to the noise library, by way of pattern values.

Currently, the noise library contains high quality, scale independent, and GPU independent hash functions, which map one or more numbers to a uniformly distributed random number in the range [0,1). These are called rand1x1, rand2x1, rand3x1 and rand4x1 in 'lib.noise'. Previously, there was no way to implement such hash functions. In order to implement this, I added bitwise operations, and functions for converting floating point numbers to and from the Bool32 data type. Since Curv is an array language, I added general support for operations on arrays of booleans. The alternative was to mimic C, add an "integer" data type, and add bitwise operations on integers. But that didn't fit in with the general philosophy of the language. So a "Bool32" is an array of 32 booleans.

More noise functions are planned. Next up is "Super Simplex Noise", an algorithm so new it hasn't been ported to the GPU yet. It claims to be faster and/or better quality than all previous implementions of Value noise and Gradient noise. We will see how it benchmarks in Curv.

The pattern library will support a new kind of graphical value called a pattern, plus a rich set of operations on patterns. A pattern is actually an "intensity field". It is a function that maps space/time coordinates onto a number between 0 and 1. Patterns can be 2 or 3 dimensional, and they can also be animated. Patterns can be used to modify or build geometry, or they can be composed with a colour map to construct a texture, which is a colour field. Once this is ready, you'll be able to type 'lib.pattern.turbulence{}', and a fractal noise pattern will appear in the preview window.

The language improvements have mostly focused on improving the imperative subset of Curv. You can now incrementally update a data structure element using the assignment operator. For example,

    a[0].foo := newvalue

There is a new statement type, called a local definition. The syntax is:

    local <pattern> = <expression>

A local definition defines one or more local variables, but with sequential, not recursive, scoping. The scope begins at the statement following the definition. Local definitions are legal wherever statements are legal. You can use them in the head of a 'do' expression, or in a parenthesized statement block (eg, the body of a 'while' statement), or in a list comprehension. This new syntax finally provides a direct translation for the local variable definitions that can occur anywhere in a statement block, as found in every popular imperative programming language. The old deprecated syntax 'var name := value' can now be replaced by 'local name = value', it is a drop in replacement.

There is now a syntax for early exit from a 'for' loop. It isn't a break statement: there are philosophical and technical objections to adding any kind of goto statement to Curv, which remains a pure functional language with simple semantics. So instead, you can now add a 'while condition' clause to a 'for' statement.

Here is what the colour function in the 'mandelbrot' demo now looks like. It nicely illustrates some of the new syntax.

    colour [x,y,_,_] =

        do

            local z = [x,y];

            local color = [0,0,0];

            local done = false;

            for (i in 0 ..< 100 while !done) (

                z := csqr(z) + [x,y];

                if (dot[z,z] > 4) (

                    local cr = (i-1)-log(log(dot[z,z])/log 2)/log 2;

                    color := [0.95+.012*cr, 1, .2+.4*(1+sin(.3*cr))];

                    done := true;

                )

            );

        in sRGB.HSV color;

At this point, the imperative subset of Curv feels like it is "done". It no longer feels clumsy to use.

I have also been thinking about simplifying the language syntax. At this point, I'm thinking, but not making backwards incompatible changes.  So far, I want to deprecate the 'where' keyword: the syntax is actually quite complicated, and it is redundant, you should use 'let' instead. I am also bothered that (a,b,c) and [a,b,c] are perfectly equivalent in all contexts: they both construct a list of 3 elements. I think that the (a,b,c) syntax is redundant, and should be deprecated.

As an experiment, I've starting changing my existing code to replace 'where' with 'let' and '(a,b,c)' with '[a,b,c]'. Everything in examples/*.curv has been changed. So far, it seems like an improvement.

[Alexandre François Garreau]

What? why [] instead of ()? () looks like functions in all other
languages, and is a very nice way to beautifully, explicitly and
«normally» present the fact to cons the arguments of a function instead of
currying them, when coming from other languages…

[Doug Moen]

Sebastien has proposed a projectional editor for Curv, where the source code is stored in memory as a syntax tree, rather than as text. The syntax tree can "projected" onto the display using a variety of syntaxes and representations, not just plain text, but text plus graphics.

What are the essential features of Curv's current text syntax that must be represented in that syntax tree, and what are the non-semantic features that can be left out? White space and indentation can be left out. The distinction between (a,b,c) and [a,b,c] doesn't carry any semantic meaning, so maybe it could be left out as well?

The same questions arise when I think about writing a code formatting tool for Curv, which reformats Curv programs to use a standardized coding style. The output of a formatting tool would be a trial run for what a program looks like when formatted by the projectional editor. Any information that is thrown out by code formatting tool doesn't need to be preserved in the syntax tree of a projectional editor. Any features of the Curv syntax and grammar that aren't used in the output of the formatting tool are candidates for being deprecated.

I've never been able to come up with a consistent set of rules for when to use (a,b,c) and when to use [a,b,c] in Curv programs, so as an experiment, I tried reformatting all of the examples to use the syntax [a,b,c] consistently, and it's fine. Nothing of importance seems to have been lost. This reduction in syntactic variability actually seems like an improvement, because now when I see a ( buried in the middle of an expression, I know it can only be used for grouping, it isn't the beginning of a list.

[Alexandre François Garreau]

What I meant is: why don’t keep *only* *curved parenthesis* (not squared
brackets). So it’s like lisp, ocaml, haskell, any programming language
with the « f(x) » notation, etc. ?

Le vendredi 6 décembre 2019, 00:03:24 CET Doug Moen a écrit :
> Sebastien has proposed a projectional editor for Curv, where the source
> code is stored in memory as a syntax tree, rather than as text.

Oh my god this is amazing, just like the old glorious days of lisp
programming environments :o before it died with GNU Emacs as the sole
survivor…

> The
> syntax tree can "projected" onto the display using a variety of
> syntaxes and representations, not just plain text, but text plus
> graphics.

I had exactly thought of this in the past (like “content/view separation”
and “stylesheets for programming languages”) :o that’s the first time I see
it mentioned anywhere else… do similar things already exist outside Curv
btw? Because I see a great patterns of good ideas united around Curv.

> What are the essential features of Curv's current text syntax that must
> be represented in that syntax tree, and what are the non-semantic
> features that can be left out? White space and indentation can be left
> out.

Well, you could “tab” stuff to improve readability *sometimes*. But that’d
better be done in the viewer.

As in regards to indentation, let me add something: do you know the
(indent n) declaration form in emacs lisp? It’s something you place in
function definitions, just after the docstring, so you indicate some sort
of “separation” after the nth argument. For instance, “map” has an
“(indent 1)” so all parameters are aligned… *but* the first (so you save
space on the screen). That way you semantically give an importance to
grouping of ordered arguments, so that indentation stays flexible and
semantically right… in an automated manner.

> The distinction between (a,b,c) and [a,b,c] doesn't carry any
> semantic meaning, so maybe it could be left out as well?

Did you separately used both for anything? why [] at first?

> The same questions arise when I think about writing a code formatting
> tool for Curv, which reformats Curv programs to use a standardized
> coding style. The output of a formatting tool would be a trial run for
> what a program looks like when formatted by the projectional editor.
> Any information that is thrown out by code formatting tool doesn't need
> to be preserved in the syntax tree of a projectional editor. Any
> features of the Curv syntax and grammar that aren't used in the output
> of the formatting tool are candidates for being deprecated.

These appears two separate questions for me. The advantage of storing
stuff as an AST is you don’t *have* to link it with a predetermined
graphical representation (so it could even improve accessibility,
ergonomy, etc.). So you wouldn’t *have* to says lists are like tuples in
haskell/ocaml/etc. and like function calling in C-like, or lists are like
C arrays, etc. you let that up to the user.

What’s great is then each one can have his own preferences on what
datatype is presented in what manner. So you wouldn’t need a textual
syntax, only a stylesheet (and standardized ways of binarily serializing
them (it could be a simple mmap of ram : not really portable but compiled
emacs do that at startup (it’s called unexec))). Then the user choose.

One cool stuff about current syntax is it *looks like* a lot like other
languages. You yourself proposed to borrow from modes for json or ocaml
for syntax highlighting and other mode-specific stuff for emacs and vim…
You see that can be useful. But not only for such shiny/fancy stuff: you
could imagine really small “programs” that could be both interpreted by a
C compiler and curv, or by javascript and curv, or by ocaml and curv…
without modification (using a subset of these). That may sound like crazy
but… think like “{\n#include "fixnum-array.csv"\n}" inside a C source code
file… it could work. You already suggested honestly to use it for syntax
coloring, why not further?

We could even imagine to do all the opposite of nowadays : instead of
writing in text, and getting it displayed as such, but parsed to be
executed, you could “write” it in binary with this editor, so that it’s
already PIC-serialized in such a way it would just work by being almost
only loaded into memory right away with mmap, and then you show that with
a stylesheet… and use that same stylesheet to generate a parser for this
language (if expressive enough a single definition can be used both ways:
for parsing and displaying).

Curv is C++ right… I’m wondering… did you think to make a declarative
parser?

> I've never been able to come up with a consistent set of rules for when
> to use (a,b,c) and when to use [a,b,c] in Curv programs, so as an
> experiment, I tried reformatting all of the examples to use the syntax
> [a,b,c] consistently, and it's fine. Nothing of importance seems to
> have been lost. This reduction in syntactic variability actually seems
> like an improvement, because now when I see a ( buried in the middle of
> an expression, I know it can only be used for grouping, it isn't the
> beginning of a list.

Oh, I see, parenthesis can already be used for something else…

Okay this is sad but understandable… You just you too want to distanciate
from lisp for easier reading…

[Doug Moen]

> Curv is C++ right… I’m wondering… did you think to make a declarative parser?

I use a recursive descent parser (procedural style). It's very easy to maintain, so I have no reason to change it.

If I ever add UTF-8 support to Curv, I'll rewrite the lexical analyser in a declarative language like re2c, because that rewrite would provide benefits.

> You just you too want to distanciate from lisp for easier reading…

What I like about Lisp is that it has a simple, consistent syntax. If I ever create a Curv 2 (with a clean break from the current syntax), then I would make the new syntax simpler and more consistent than it currently is.

However, Lisp syntax is missing some important properties that I think Curv should have:

* Infix operators: a * b + c - d.
* An infix '.' operator, A.B, for selecting a name from a namespace. It's important that A is a general expression, and that you can chain the dot operator and write A.B.C.D.
* Curried function call syntax, 'f x y z'. The equivalent in Lisp, (((f x) y) z), is so terrible to use that you can't use curried functions in Lisp the way you do in Curv.
* A common feature of all the above syntaxes is *chaining*: they allow you to chain a sequence of operations together without having parentheses pile up at one end of the expression.

[Alexandre François Garreau]

Le vendredi 6 décembre 2019 01:49:32 CET, vous avez écrit :
> > Curv is C++ right… I’m wondering… did you think to make a declarative
> > parser?
> I use a recursive descent parser (procedural style). It's very easy to
> maintain, so I have no reason to change it.

Wouldn’t extend that to be declarative may be, maybe, easy to maintain
too?

> If I ever add UTF-8 support to Curv, I'll rewrite the lexical analyser
> in a declarative language like re2c, because that rewrite would provide
> benefits.

Mmh.

> > You just you too want to distanciate from lisp for easier reading…
>
> What I like about Lisp is that it has a simple, consistent syntax. If I
> ever create a Curv 2 (with a clean break from the current syntax), then
> I would make the new syntax simpler and more consistent than it
> currently is.

Not more like lisp? or wisp or something alike? or more integrated with
guile and/or libfive?

> However, Lisp syntax is missing some important properties that I think
> Curv should have:
>
> * Infix operators: a * b + c - d

Wisp and some others have this. Also I found some implementations of this
as a macro (like “expr” in tcl/sh).

> * An infix '.' operator, A.B, for selecting a name from a namespace.
> It's important that A is a general expression, and that you can chain
> the dot operator and write A.B.C.D.

yeah, the lisp is (D (C (B A)))… it looks like curried functions as you
say it later…

But it could be emulated with macros. Actually, I could even think of
something interesting: using (common lisp) implementation of namespaces to
do this. So it becomes A:B:C:D. Having something like curv, where each
one is actually an alist, would be damn interesting…

> * Curried function call syntax, 'f
> x y z'. The equivalent in Lisp, (((f x) y) z), is so terrible to use
> that you can't use curried functions in Lisp the way you do in Curv.

A macro, or even a function, doing this could be possible… It recalls me
a bit the *variadic* version of the “compose” function (so (f (g (h x))) =
((compose f g h) x)) but associative the other way…).

But I have to say I (and a lot of other lispers) am not fond of currying…
the only usage I find is trivial partial evaluation… but it would be even
more neat to do arbitrary partial evaluation on whatever the arguments
whatever their order…

Though I must say Curv awoken a lot of aesthetic attraction to currying.

> * A common feature of all the above syntaxes is *chaining*: they allow
> you to chain a sequence of operations together without having
> parentheses pile up at one end of the expression.

it’s trivial to make lisp macros like the « | » pipe in bash (or your >>
operator)… but I never saw anyone doing this… probably because it’s not
really idiomatic and often programs are not linear enough?