[ANN] Dunaj project, an alternative core API for Clojure

[Jozef Wagner]

I'm happy to announce a project called Dunaj [1], which provides an alternative core API for Clojure. Its main aim is to experimentally test major additions to the language. 

Dunaj /ˈdunaɪ/ is a set of core language experiments aimed to improve Clojure language and its core API. It deals with language features that require changes across different parts of Clojure and which cannot be evaluated in isolation. Dunaj aims to bring Clojure even more towards simplicity, consistency and performance. 

It is intended to be used by regular Clojure developers, either for application or library development.

Dunaj was created to test 10 experiments that bring significant changes to the Clojure language. As there is a substantial number of additions and changes, I want to try a bit unconventional approach here. Before I'll release the actual library, I will introduce Dunaj's experiments in a series of individual posts. Every part states the motivation behind the experiment, introduces changes and additions to the language and demonstrates its intended use. If you do not want to miss any of this, you may want to register for a mailing list at [1] or follow @dunajproject at Twitter.

-- Jozef Wagner

[1] http://www.dunaj.org/ 

[adrian...@mail.yu.edu]

Not sure exactly how to properly express my impression here, but it strikes me as odd that this project would not come out of direct collaboration with Clojure's core contributors. Proposed features should probably go through the official channels if you are serious about getting them included in the language. 

[Ben Wolfson]

Why? One of the purposes of PyPy, for instance, was to make it easier to experiment with new features in possible Python interpreters, and that's a separate project not undertaken in direct collaboration with CPython's authors.

--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojure+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Ben Wolfson
"Human kind has used its intelligence to vary the flavour of drinks, which may be sweet, aromatic, fermented or spirit-based. ... Family and social life also offer numerous other occasions to consume drinks for pleasure." [Larousse, "Drink" entry]

[Michael Klishin]

On 6 March 2015 at 00:45:47, adrian...@mail.yu.edu (adrian...@mail.yu.edu) wrote:
> it strikes me as odd that this project would not come out of
> direct collaboration with Clojure's core contributors.

I should point out that there's enough people in the community who
do not find Clojure's core contributors, ahem,
exactly open to collaborating
 on stuff not seen as exciting or important in their circle.

> Proposed
> features should probably go through the official channels if
> you are serious about getting them included in the language.

Attempts to introduce major changes to clojure.core have incredibly low
chances to succeed for reasons that are outside of control of Dunai's author.

A completely new 3rd party project has the same low changes of being
*widely* adopted but may influence clojure.core in some areas in the long run
and be useful to some today. Or the day it stops being vaporware with a mailing
list and no source code publicly available.
--
@michaelklishin, github.com/michaelklishin

[Alex Miller]

I'm happy to see experiments if we can learn something useful. Can't really judge more till the posts are out. Seems perfectly possible that something promising could get a design page and move towards inclusion in some way.

Alex

[Alex Baranosky]

Yeah, I'm excited to see some of the 10 write-ups. What's the ETA on the first one?

On Thu, Mar 5, 2015 at 6:02 PM, Alex Miller <al...@puredanger.com> wrote:

I'm happy to see experiments if we can learn something useful. Can't really judge more till the posts are out. Seems perfectly possible that something promising could get a design page and move towards inclusion in some way.

Alex

[Ivan L]

Just a quick glance at the example project shows integrated type definitions.  I'm curious for sure.

[Alex Baranosky]

Where is this example project?

On Thu, Mar 5, 2015 at 7:40 PM, Ivan L <ivan.l...@gmail.com> wrote:

Just a quick glance at the example project shows integrated type definitions.  I'm curious for sure.

--

[sesm]

Ivan was probably talking about dunaj-starter repo on github:
https://github.com/dunaj-project/dunaj-starter/blob/master/src/dunaj_starter/core.clj

пятница, 6 марта 2015 г., 9:42:00 UTC+3 пользователь Alex Baranosky написал:

[Jozef Wagner]

I'm planning to introduce experiments every other day, starting later today.

[Jozef Wagner]

Experiment #1

*Deconstructing Core API*

While Clojure provides its functionalities in multiple namespaces (e.g. clojure.string, clojure.zip), the majority of it is defined in a single namespace called clojure.core. The first Dunaj experiment explores the idea of having multiple small namespaces where functions, macros and other public vars are grouped by their purpose. It investigates whether such separation is possible at all and whether it can be practical and useful.

The goals of the first Dunaj experiment are as follows:

- Devise a new user centric core API comprising multiple namespaces, leaving bootstrapping and low level vars in the clojure.core.

- Define a concept of API presets that control which functions, macros and vars gets referred by default.

- Let user choose which API preset he/she wants to use in his/hers namespace, using classic clojure.core as a default.

The upside of this approach is that backwards compatibility is maintained and users can freely intermix multiple APIs in their projects. Functionalities can be more logically separated by their purpose. List of automatically referred vars are no longer driven by the namespace in which they were defined, but this list is handled by separate API presets that can be extended and customized.

You can read more about this experiment at http://www.dunaj.org

-- Jozef

[Bozhidar Batsov]

You might want to add some commenting capabilities to those articles. They'll be more valuable in the context of discussions IMO.

[Jozef Wagner]

Thank you. Based on your suggestion I've added disqus commenting there. 

[Jozef Wagner]

While the first experiment took long to implement, it was fairly easy to understand and I think we can move on to the second one.

Second Dunaj experiment adds support for optional type signatures that can be used to annotate functions, protocol methods, type fields and local bindings. These type signatures are then used to automatically generate host type hints. At run-time, type signatures are stored in vars' metadata, available for third party type checking, data validation and documentation tools.

Goals of the second Dunaj experiment are as follows:

- Provide developers with a means to document type signatures for functions, protocol methods, let-like bindings and deftype fields.

- Automatically generate type hints from provided type signatures, including primitive ones.

- Open up type signatures for custom extensions and use in third party data validation and type checking tools.

Dunaj aims to decomplect type declarations from type checking tools. It provides conventions and syntax extensions for defining type signatures, while leaving their exact interpretation to other libraries. Existing protocols, types and host classes can all be used as type declarations. By generating type hints automatically, users do not have to write types twice (first time as a type signature and second time as a type hint). Specifying type signatures does not produce any run-time overhead, as all processing is done at a macro expansion time.

As I don't want to write long emails here, you can read more about this experiment at http://www.dunaj.org .

Best,

Jozef

[Jozef Wagner]

Third Dunaj experiment: Protocols First

 

One of Dunaj’s objectives is to facilitate host independent style of programming. The third Dunaj experiment aims to transform core API into one that uses protocols for all of its abstractions. Additionally, Dunaj introduces a concept of a factory protocol that represents an abstraction of a constructor for an abstract data type. Protocols and protocol methods are part of the Dunaj’s interface that is oriented towards implementers of custom types and language extensions and they are documented separatedly in the SPI part of Dunaj’s documentation.

Goals of the third Dunaj experiment are as follows:

 

- Provide protocols for all existing built-in abstractions that were in Clojure specified by host interfaces.

- Modify existing core functionalities to build upon protocols instead of specific types, host classes or interfaces.

- Introduce a separate representation of deftypes, treating the underlying host class as an implementation detail not automatically imported into the current namespace.

- Introduce a new idiom of factory protocols, which are used to represent an abstraction of a constructor for an abstract data type.

As usual, you can read more about this experiment at http://www.dunaj.org 

Regards,

Jozef

On Thursday, March 5, 2015 at 10:33:53 PM UTC+1, Jozef Wagner wrote:

[Dave Sann]

very interesting work and well presented, keep going.

[Jozef Wagner]

Having modern Clojure features at hand, and free from having to support old

versions and existing application code, Dunaj has more freedom to test new

things and can take more liberal and fresh approach.

Next experiment is a short one, a bit opinionated, and it mainly deals with how Dunaj designs its public API.

Experiment #4: Towards Simpler API

With core API splitted into separate namespaces and with protocols being used to specify core abstractions, Dunaj further simplifies its API by providing and adhering to additional idioms, conventions and best practices. Moreover, Dunaj changes how compiler recognizes special symbols and makes them fully qualified.

You can read more about this experiment at http://www.dunaj.org 

Jozef

On Thursday, March 5, 2015 at 10:33:53 PM UTC+1, Jozef Wagner wrote:

[Jozef Wagner]

With today's experiment, we are going to be halfway through Dunaj. I hope you are finding these write-ups interesting and

that eventually, they will enable us to write even more powerful and performant Clojure programs. Thanks for all the response so far.

If you haven't read the previous write-ups yet, please do it now. The order was not chosen randomly and following experiment relies on the existence of previous ones.

Experiment #5: Reducers First

Reducers were introduced in Clojure 1.5 and in version 1.7, transducers will be added to the language. Reducers are however a completely optional feature and collection related functions still return (mostly lazy) seqs by default. Dunaj makes reducers a new default and deemphasizes the role of seqs by making them an optional feature for most of the core API.

Dunaj’s initial and the main vision is to create API that uses reducers by default and to provide additional reducible data sources and consumers of reducible collections. The overhaul of reducers and collection related API is Dunaj’s biggest experiment. Goals of this fifth described experiment are as follows:

- Transform collection related API into one that uses reducers by default

- Enhance reducers to support straightforward conversion to lazy sequences

- Provide full API support for transducers

You can read more about this experiment at http://www.dunaj.org 

Best,

Jozef

On Thursday, March 5, 2015 at 10:33:53 PM UTC+1, Jozef Wagner wrote:

[Jozef Wagner]

Experiment #6: Enabling Host Performance

Clojure provides excellent facilities for creating, composing and building on abstractions. Thanks to great interop support, it is easy and even idiomatic for

 performance sensitive code to fall back into host’s primary language. Dunaj offers additional and fully optional ways to embrace the host platform from the

 comfort of your favorite Lisp. This experiment is specific to the JVM host, and provides following enhancements:

- Improved support for arrays, providing array managers and an optional immutable collection facade on top of arrays

- A dedicated namespace called dunaj.host.int that implements macros for working with int values without unnecessary promoting or boxing

- More primitive types for function arguments and return values

- Low level data containers for bulk data processing

You can read more about this experiment at http://www.dunaj.org

Best,

Jozef

On Thursday, March 5, 2015 at 10:33:53 PM UTC+1, Jozef Wagner wrote:

[Jozef Wagner]

From now on, the experiments will tackle more practical issues and

will add completely new features to the language. There will be less

theory and more examples.

Experiment #7 - Data Formatters

A conversion between different data formats is a frequent operation

in many applications. Dunaj introduces the concept of data formatters

that formalizes the process of converting data from one format into

another.

A formatter is a collective name for Dunaj’s data parser and printer.

A parser converts information from its external or low level

representation (used for storage or communication) into form that is

of higher level of abstraction, and is better understood and supported

by the application. Printer does the opposite thing. Formatters are

not limited to parsing from/printing to strings, but are designed to

work with any type of data, even for binary decoding/encoding.

Goals of this experiment are as follows:

* Provide dedicated functions for parsing and printing that integrate

  well with the rest of the API, including transducers

* Provide protocols and helper functions for implementers of custom

  data formatters

* Make formatters efficient by utilizing available optimizations

You can read more about this experiment at http://www.dunaj.org 

Regards,

Jozef

On Thursday, March 5, 2015 at 10:33:53 PM UTC+1, Jozef Wagner wrote:

[Herwig Hochleitner]

2015-03-12 12:15 GMT+01:00 Jozef Wagner <jozef....@gmail.com>:

Experiment #5: Reducers First

Wow! There are some great ideas on how to unify reducers with lazy-seq.

Have there yet been thoughts on how reduce-kv could fit in a reducer stack?

Obviously, the notion of a transducer could be extended to allow distinct input- and output arities, which would fit into reducing-fn arities > 2.

But could this be made to interact smoothly with tuple-taking and -producing functions?

I've done some exploratory programming on this in the past [1], and it turned out a pretty decent map-handling lib, that comes in handy frequently. Still, only the formulation of transducers opened my eyes to the possibility of efficient multiple-return-values by using transducers in the style of reduce-kv.

kind regards

[1] https://github.com/webnf/webnf/blob/master/base/src/webnf/kv.clj

[Jozef Wagner]

Yes and yes. In Dunaj this is called the unpacked reduction, and there is IUnpackedRed protocol that is used by coll implementations to provide reduce-kv like functionality. Please see this example on how Dunaj handles reductions with >2 arguments. No intermediate tuples are created when using unpacked reduction...

https://www.refheap.com/98510

Jozef

[Jozef Wagner]

Another big experiment for today. As new reducer features

(continuations) will be applied heavily here, please read

5th experiment if you have not done so yet. 

Prepare for the longest write-up so far.

Experiment #8 - Computer and Network Resources

Most applications perform some file and networking I/O.

Clojure leaves this domain largerly uncovered and advises users

to use host facilities instead. Dunaj attempts to cover this realm

with an extensive and powerful set of features dedicated to computer

and network resources.

Goals of this experiment are as follows:

* Design and implement the concept of resource scopes

* Develop facilities for implementers of computer and network

  resources

* Create abstractions for basic resource functionalities and

  integrate them with the rest of core API, mainly with collections

  and transducers

* Provide implementation for widely used resources

* Devise abstraction for composing resources into 'systems'

You can read more about this experiment at http://www.dunaj.org 

Regards,

Jozef

On Thursday, March 5, 2015 at 10:33:53 PM UTC+1, Jozef Wagner wrote:

[Henrik Eneroth]

Amazing work Jozef. Very interesting experiments.

[Jozef Wagner]

Thank you for all the responses so far!

9th experiment is a small one compared to previous ones, but still worth of mention.

Experiment #9 - Improved Math Facilities

Dunaj extends available math facilities with means to specify precision

of arithmetic operations and to round numbers. A pluggable mechanism for

using random number generators is introduced, and math related API was

extended with functions of an angle, exponentiations and logarithms.

Instead of using prefixes and suffixes like in Clojure, facilities for

unchecked and precise math are in Dunaj specified in separate namespaces.

Common arithmetic functions are provided in dunaj.math namespace, with

unchecked functions and arithmetic functions with arbitrary precisions

defined in dunaj.math.unchecked and dunaj.math.precise respectively.

List of available math functions is extended with exponentiation,

logarithms functions and several miscellaneous ones. A separate namespace

called dunaj.math.angle contains math functions of an angle, with

floating point numbers. Precision of these functions is as implemented by host.

Both circular and hyperbolic functions are available.

You can read more about this experiment at http://www.dunaj.org 

Best,

Jozef

On Thursday, March 5, 2015 at 10:33:53 PM UTC+1, Jozef Wagner wrote:

[Jozef Wagner]

Dunaj, an alternative core API for Clojure, has been released! 

Try it yourself and check out its extensive documentation at http://www.dunaj.org/

Last Dunaj experiment aims to improve API's primary documentation.

Official documentation is available at project's homepage at

http://www.dunaj.org. API and SPI documentation is

automatically generated from Dunaj sources. Other parts of

the documentation are written in AsciiDoc format and are available in

project's repository alongside other sources.

Everything is released under the same license as Clojure.

Library used for the actual documentation generation can be used

to generate documentation for any Clojure project, and will be

released soon.

API documentation is presented in a clear and user friendly way.

Vars in a given namespace are grouped by their purpose into

several categories. In addition to Var's docstring, type signatures,

examples and see also references are provided.

Moreover, Dunaj defines new metadata keys for vars that provide

additional information to IDEs. With these metadata, editors can

offer improved and more precise highlighting and indentation.

You can read about this and previous experiments at the documentation

section of Dunaj at http://www.dunaj.org/doc.html 

Best,

Jozef Wagner

On Thursday, March 5, 2015 at 10:33:53 PM UTC+1, Jozef Wagner wrote:

[James Trunk]

Congratulations on the release!

Thank you for putting yourself and your ideas out there in this way, and for reminding us not to settle for what we have now, but to continually push towards ever greater heights.

Best,

James

[Jozef Wagner]

Thank you James!

If someone has already tried Dunaj, I'd be interested to know whether you've encountered some stopping bugs, mainly on platforms other than Linux.

Best,

Jozef