Welcome to this year's first issue of Niteration, the newsletter for the Nit project.
Combined changes for November and December 2015 and January 2016:
Have contributed (or co-authored patches): Alexis Laferrière, Jean Privat, Alexandre Terrasa, Lucas Bajolet, Philippe Pepos Petitclerc, Romain Chanoir, Frédéric Vachon, Alexandre Blondin Massé, Guilherme Mansur, Jake Russo and Denis Martinez.
The v0.8 milestone marks, among other things, the 10.000 commits in the main repository.
Version v0.7 was released more than a year ago, since then there was a lot of improvements:
new
factories, with
statement, improved constructors, conditionnal importation, multi-iteratorsTo celebrate, we regenerated the primitive compiler that lives in c_src
and that is used by the initial make
, the previous regeneration was Wed Jun 10, 2015.
In order to help people to reach the community and ask questions, we opened some gitter rooms:
New packages are provided for popular text editors:
A first attempt to experiment with Nit on Windows was documented. Nothing fancy yet but a first step in the right direction.
November, 28th 2015, we organized a hackathon in Montreal.
The topic was nitcorn, our lightweight framework for developing web applications.
A dozen of developers and enthusiasts met to improve the framework and web applications. The following presents most of the feature developed.
The core or the framework provides new services: a logging facility, a simple reverse-proxy route, a htcpcp server and a RESTful API generator.
This last feature is based on a tool that generates a Nit module which implements Action::answer
to redirect request to a static Nit method. It checks the presence of args, their types, deserializes objects as needed and calls the target method.
Missing arguments or arguments with errors (wrong type, failed deserialization, etc.) are replaced by null
when the corresponding parameter is nullable
. If the parameter is non-nullable, or if there is any other error, the generated code calls super
from answer
for the user code to handle exceptions and errors.
With the restful
annotation we can write a normal method with static types such as: (from the example)
# User code
class MyAction
super RestfulAction
# Method answering requests like `foo?s=some_string&i=42&b=true`
fun foo(s: String, i: Int, b: Bool): HttpResponse is restful do
var resp = new HttpResponse(200)
resp.body = "foo {s} {i} {b}"
return resp
end
...
end
The remaining changes in the core are mainly bugfixes.
CurlHTTPRequest
f39e001answer
allowing more control and revamp the README file b12bfddNitiwiki is a static wiki compiler.
A new trail
directive will register the target page as an element of a trail. trails
are chained together and will display the navigational links prev
for the previous page of the trail, next
for the next page of the trail and up
to go to the pages that has used the trail
wikilink.
For instance, if the page doc.md
has the following content:
To use nitiwiki, first [[trail: install|install it]],
then [[trail: simple_wiki|create a first wiki]].
You can also do advanced things like:
* [[trail: github|editing pages with github]]
* [[trail: templating| adapting the templates]]
A trail will be made, and will consist of the sequence of pages install
, simple_wiki
, github
and templating
. On each one of these pages, there will be links for the previous, the next page and the doc.md
page.
If a page includes trail wikilinks and is also the target for trail wikilinks, then the two trails are merged and pages will be visitable in a depth-first order. This nesting of trails can be used to have sections and sub-sections.
See trails in action: http://nitlanguage.org/manual/
While nitiwiki is a static generator, it can now be connected with a simple nitcorn server to publish the static files generated by nitiwiki and allow modifications from a web form. It is already applied on http://xymus.net/ to manage the main page.
The web server reads the config.ini
of the target to find the path to the public files. It writes the new markdown to the source folder and regenerates the wiki on each modification. The modification forms apply the template of the current wiki. (with some imperfections)
There is also a basic password authentication system using a list of hashed passwords in a simple text file. It should be enough for simple deployment of the server and this file can be ignored by git.
See nitcorn+nitiwiki in action: http://xymus.net/
Tnitter is simple web-application used as an example of nitcorn.
A RESTful API was added to the server and an Android client was developed as a proof of concept.
NitRPG is a gamification of development on github.
The main change is the use of mongodb to store the database (instead of plain JSON files).
mongodb
instead of json_store
6c74d9bA new tool, nitweb, offers navigation through the model and the source code of nit programs with a web browser.
nitweb uses nitcorn actions to provide an API to the model and source code. It can understand simple requests and provide answers in simple HTML or in JSON.
See nitweb in action: http://nitweb.nitlanguage.org/
app.nit is a framework for portable applications for Linux, Android and soon iOS.
gamnit is a portable game and multimedia framework based on app.nit
There was a lot of work on these two frameworks during the last months.
New core modules are available, with a lot of features:
textures: Provides portable services to load textures. Textures are organized as trees where only the roots have the pixel data, and subtextures relay load calls to the root. This structure should remain as is but more features need to be implemented, namely auto reloading textures when invalidated and services to unload the textures.
programs: Structures to organize GL programs and shaders and to facilitate access to shader variables. The main advantages of this API are static types on shader variables and the gamnit programs that can be specialized.
cameras: Add an abstract Camera class and a simple EulerCamera based on euler angles, perfect for FPS type games or top-down strategy games. This camera's main feature is to produce the MVP matrix.
More cameras should be implemented, they should also be considered objects attachable to a scene when we have scenes. The EulerCamera lacks some feature, like the usual look_at
.
flat: a new API meant to replace mnit for writing 2D games. Client modules only have to create instances of Sprite
which will be displayed as long as they are in App::sprites
.
At this point, the flat
API is not optimized and you may see a slowdown at the insertion of new sprites. However, we could improve the performance by preallocating buffer space for more sprites, and by caching sprites state on the GPU to avoid unnecessary updates.
Asteronits: a game used as an example for the flat
API. In this game, the player controls a ship, avoids asteroids and destroys them. The gameplay and UI is limited by design to be a simple example. Note that the touch_ui
variant is used on Android only but it also works on Linux.
texture_atlas_parser: a tool that reads XML TextureAtlas files and produces Nit classes with one attribute per subtexture. We could move this tool out of Asteronits if it is useful for any other projects.
model_parsers: parsers to load 3D models from the ASCII based format of .obj and .mtl files. Also as support intro a new kind of asset, app::TextAsset
, with lazy loading in the style of Sound
and Texture
. Model parsing services are used by the 3D API of gamnit to load 3D models generated by modeler tools like Blender.
The services in gamnit::model_parsers
are independent of gamnit. However since they are game related, it is a nice package to classify them in. They still can be used by any other 3D graphics engine, as gamnit::network
which is also independent but game oriented. This adds a small duplication in their representation because a model is stored differently once loaded in gamnit. But the format in gamnit is so low-level (only arrays of Float) that we need an intermediate representation to triangulate faces and stuff like that.
depth: a framework for portable 3D game development in Nit.
This framework is based on a list of Actor
, in app::actors
, which are drawn at each frame. Each actor is composed of a model and other information specific to this instance: position in the world, rotation and scaling. Each Model
is either a composite of models or it is composed of a Mesh
defining its geometry and a Material
defining how to draw the model. Material
can be subclassed to use custom shaders. These main classes are defined in the module depth_core
.
gamnit depth is built upon app.nit and it works well with gamnit flat. Clients are portable across GNU/Linux and Android, andgamnit flat can still be used for UI elements.
vr: support for VR to the gamnit depth framework. By importing gamnit::vr
, all drawings are duplicated for a stereoscopic view and the world_camera
orientation is updated from head tracking data provided by the Google Cardboard SDK.
VR is applied to a variant of the model_viewer
app for testing only. The UI should still be updated for a nicer VR experience.
This is done by simple refinement, which allows for easy (but dirty) conversion to VR using nitc ... -m lib/gamnit/depth/vr.nit
. However this does not allow to easily switch between VR and a classic view. This could be improved as needed in the future by using a subclass of EulerCamera
and a few conditions in frame_core_draw
.
This framework is far from complete at this point, however it is designed for extensibility.
model_viewer: A simple 3D model viewer, used as basic example for the gamnit depth API. It displays basic geometric models with the included materials, as well as models loaded from files and a globe of the earth, a custom model with a custom material.
Tinks! 3D: built with gamnit depth. It uses the same logic as the 2D client so they are compatible for multiplayer games. To support this client there was a few fix to gamnit and some new features, most notably a customizable particle system with two implementations, one for smoke and one for explosions.
At this point, we wrap most of the OpenGL ES 2.0 features.
app.nit gains a series of portable services to execute HTTP requests asynchronously from graphical programs.
The HTTP request services are very simple by design, it is not an attempt to define a true API to build the request. It is currently limited to GET calls to a simple URI, for example a simple use may look like:
print "http://xymus.net/rest/list?query=asdf".http_get.value
The services are:
Text::http_get
makes an HTTP request and blocks until a response is received. It returns HttpRequestResult
, a subclass of MaybeError
, with a possible error, status code and response body content. This service is implemented independently on each platform, using GDK + Curl on GNU/Linux and Apache HTTP client services in Java on Android.
App::run_on_ui_thread
sends an instance of Task
to be executed on the main UI thread when possible.
AsyncHttpRequest
combines the two previous features to execute an HTTP request asynchronously, deserialize the result from JSON (if needed) and execute custom behaviors on the main UI thread. Users of this service should subclassAsyncHttpRequest
and implement as needed before
, on_load
, on_error
and after
. By default, all user code is executed on the main UI thread and as such users do not have to worry about threading logic.
android.nit is the platform support for Andoid.
The Task
interface defines a fragment of Nit code to execute in its main
method. This interface should be compatible between different platforms and parallelization engines.
NativeActivity::run_on_ui_thread
is widely used on Android as most logic is executed on background threads but only the UI thread can modify the UI. It is a perfect example of how the Task
interface can be used.
Task
and use in for Android's run_on_ui_thread
536a3ffUse the Android Java API to load and decompress image files, and remove the old use of a local copy of libpng. This new system is much more simple, produces smaller APK files (~30% smaller) and supports more image formats, including jpeg. It relies on the Java FFI to access the API and the C FFI to move pixels around.
Adds a missing feature to the Java FFI: pinning references to Nit objects from Java. Both services, ..._incr_ref
and..._decr_ref
, are "simple" redirections from Java to the nitni service in C through the JNI.
We also tweak basic Nit features on Android so it is more practical on the short-middle term.
It is better to not handle signals on Android so that it falls back on the Android stacktrace, which is pretty useful for debugging. We might want to handle the signals once we successfully integrate libunwind to the Android apps.
Dropping MIPS support saves some compilation time. The related commit can be reverted when supporting MIPS is useful.
We deactivated the GC on Android as it is currently broken. Unexpected behaviors happen more frequently when using the Java FFI in a threaded app. There are possible solutions to this problem; newer versions of libgc are tweaked for Android, better compilation configs may help, and double checking the Android lib for objects that should be pinned for use from Java...
Again a bunch of performance improvements related to the manipulation of Unicode strings.
We wanted a way to detect and possibly remove a prefix from a string but there was no straightforward way. Se we proposed one:
var str = "hello world"
var p = str.prefix("hello")
assert p != null and p.text_after == " world"
assert str.prefix("fail") == null
Same with suffix
Various updates to Bytes
, added a new escaping method taking into account \u and \x escape sequences. This added string-like functions to Bytes, along with an abstraction for Byte-based patterns
A lot of unnecessary unboxings were done when comparing bytes in FlatString, this was fixed by short-cutting the access to the bytes of a FlatString via _items
. Also, the operation is now optimized when working on any FlatText
Update of the grammar to support the enhancement of literal strings and chars via prefixes.
The package buffered_ropes
was removed since it was useless, experimental and did not bring anything but woes to the maintainer of the text
package.
buffered_ropes
package 7ab27c3A batch of brand-new optimizations on String
, especially regarding the manipulation of UTF-8 characters.
As a baseline, for the JSON parser on a 30MiB json file, the execution took approximately 10 seconds for ~38 GIr with Valgrind, now, the total runtime is less than 2 seconds for ~11 GIr with Valgrind.
is_hexdigit
service on Char
a898a5dSome specific optimizations in the libraries were performed.
We factorized and improved some code related to the Array (and fixed a bug). A NativeArray::memmove
method was also added, but without an intern implementation for boostrap compatibility reasons.
The improvements should mainly concern the clients of shift
, unshift
and insert
(not that many in fact). Using the new test bench_seq (with only the array and an argument of 5000)
Before: 0m1.828s After: 0m0.292s (-84%)
lib/core
was extended with CircularArray
for an array with efficient push/pop/shift/unshift.
Some number with bench_seq. Run valgrind with n=1000
push (Ir/call):
pop:
unshift:
shift:
insert:
We optimized the nitcc_runtime library and the generated code. Numbers with raw parsing of a 30MB json file from nitrpg (only the AST creation, no further analysis)
There were also other optimizations.
A few new things in the core library that are not related to performance or strings.
Collection::sample
645bf13We now provide the services to_json_string
and from_json_string
as these might be useful for simple scripts. Both services report errors only on the console. This behavior is enough for simple scripts but more complex programs should still useSerializer
and Deserializer
services.
We added a simple and faster ad-hoc JSON parser working in a similar way as DOMXmlParser
or SExpParser
. The old nitcc parser is still available by default.
To track regressions, we added a JSON parser benchmark between different languages and Nit using 3 variants:
FlatString
Concat
and FlatString
Other than that, we do as well as Go, which is nice.
Some work on the mongodb wrapper. Mainly bug-fixes.
skip
and limit
when calling find
b3cc7d2Some work on the github API wrapper. Mainly bug-fixes.
Experimentations with parallel Nit are progressing slowly, now there is a proof of concept of joinable task and threadpools. A JoinTask is joinable, meaning it can return a value and if the value is not set yet, it blocks the execution. A ThreadPool just register and execute tasks .
console gets a simple progress bar that refreshes itself.
import console
var pb = new TermProgress(10, 0)
for i in [1..10] do
pb.update(i)
end
Will display something like:
20% [==========================> ]
missing-doc
warningsThe missing-doc
advice is now conditioned to the presence of a doc on the module.
The rationale is that if the module is documented, then the programmer does care and we show places where the documentation is missing, else the programmer does not care then we do not talk about documentation in order to not alienate him.
missing-doc
advice is conditioned to the presence of doc on the module f251627The handling of alternative of intern methods is improved since an extern body can be used as a fallback.
redef class Int
fun foo is intern `{ return foo(self); `}
end
as
casts used as typeThe as
construction can be used to implicitly type class attributes. When an expression setting an attribute ends with a cast such as.as(String)
, we can safely define the type of the attribute as String
.
class A
# Before
var s: String = data_store["s"].as(String)
# After
var s = data_store["s"].as(String)
end
Since as
casts are unsafe by definition, it is not recommended to abuse this feature.
as
casts to implicitly type class attributes 9444d64The parser (and the AST) accepts qualified class identifiers. Now it can parse:
redef class m0::A
super m1::S
redef type m2::B::T: Int
fun foo: m3::G[m4::C] do end
end
Currently, the class qualifiers, if any, are just ignored by semantic phases. Like with method qualifiers.
A very borderline issue, when a signature contains more than one vararg parameter, was solved. They are still refused in the common case but can occurs when multiple initializers are combined into a single constructor.
The implemented semantic is the following:
class A
fun x(i: Int...) is autoinit do end
fun y(j: Int...) is autoinit do end
end
var a
a = new A(10, 20) # OK: i=[10], j=[20]
a = new A(10, 11, 20) # Refused
a = new A([10, 11]..., 20) # OK: i=[10, 11], j=[20]
a = new A([10, 11]..., [20, 21, 22]...) # OK: i=[10, 11], j=[20, 21, 22]
a = new A([10, 11], [20, 21, 22]) # Refused but a hint is given that `...` may be missing
Constructors have a long history of attempted specifications, therefore a lot of complex cases and code was still present to avoid breaking things. We simplified the code of auto_super_init
that deals with injecting implicit calls to various semantics of super
in a constructor. Now, there are only two remaining cases:
super
(polymorphic and linearized) is injectedinit
is injected iff there are no conflict and no arguments.More work on constructors are still expected.
There were a lot of ways to visit and work with the AST. Working with the model was always complex because there are many ways to do it without anything standard or considered as a good practice.
Recently, the model_visitor
provided a simple way to walk through the various elements of a model. This one was improved and new filters were added to accept/reject entities without documentation, attributes and test_suites.
Moreover, a new class, ModelView, offers configurable views to a model.
A lot of model clients (doc, uml, metrict) were also updated to use the new ModelView and ModelVisitor.
A proof of concept of a Static Analysis Framework was developed. In fact the code is quite old, but the integration with the master branch did take time.
Currently, only the basic behaviors of a StaticAnalysis and ForwardAnalysis are introduced. A ReachingDefAnalysis is provided as an example of StaticAnalysis implementation.
Now all files in tests (including alts), except one, do not make nitpick
crash. This should improve the quality of error messages in vim.
try_get_primitive_method
d00bd4dNit is an expressive language with a script-like syntax, a friendly type-system and aims at elegance, simplicity and intuitiveness.
The Nit project is about the development and experimentation with the Nit language, its tools, its libraries and its applications.
Niteration is a pseudo-monthly newsletter about the development and the progress of the Nit project.