[Niteration] October 2015

[Jean Privat]

Welcome to this year's seventh issue of Niteration, the newsletter for the Nit project.

Combined changes for October 2015:

• changes: v0.7.8 .. v0.7.9
• shortstat: 201 files changed, 4449 insertions(+), 1554 deletions(-)
• Pull Requests: 43
• non-merge commits: 166

Have contributed (or co-authored patches): Jean Privat, Alexandre Terrasa, Alexis Laferrière, Lucas Bajolet.

Highlight of this month

Multi-iterators

Introduce multi-iterators in for.

A single for can now iterate over more than one collection at once. An iterator is used for each collection and the iteration is finished once the shortest iterator is finished.

for i in [10, 20, 30], j in [1..10] do print i+j # outputs 11 22 33

As expected by POLA, multi-iterators are also naturally usable on maps and comprehension arrays

var m = new HashMap[Int,String]
m[1] = "one"
m[2] = "two"
for k, v in m, i in [1..10] do print "{k}:{v}:{i}"
# outputs 1:one:1 2:two:2

var a = [for i in [10, 20, 30], j in [1..10] do i + j]
print a # outputs [11, 22, 33]

• Merge: Multi iterators 49c3a1a

Package autonomy

Introducing autonomous packages, they should be coherent and work as is without specific user hack.

The fixed issues are:

• The absence of required source files (eg. because generated by nitcc or nitserial) that make tools unusable or report errors. The proposed solution is a new optional Makefile goal pre-build that has to generate the missing files. close #1720

• The requirement of arbitrary -I that make the tools (eg the vim plugin) unusable on the projects without prior knowledge. The only project that uses a rogue -I is online_editor that require things from nitc. The proposed solution is to virtually move src to contrib/nitc as a symbolic link. New project could then choose to import nitc (or parts of it) without being in src.

• The presence of broken nit files. Currently, only 3 broken files remain: contrib/pep8analysis/src/parser/parser_abs.nit,contrib/nitc/parser/parser_abs.nit and contrib/nitcc/src/nitcc_lexer0.nit. These are still open issues and will be dealt with later.

• Merge: Rationalize the usage of tools on packages deaa490

Improvement in Tools

Robustness

A lot a bugfixes to the tools where commited this month. Most of them concern unchecked codepath with some rare options. Others are related to robustness.

The main improvement is a is_broken flag on all entities that simplify the client's code to check the validity of entities before working with them.

Another important change is the use of bottom types to indicate broken or inconsistent types. This last change makes the whole model more robust since bottom types are sometimes returned instead of aborting.

• Merge: src: fix null receiver error on useless-signature warning c73b0f8
• Merge: More robustness in keep going 3740ae5
• Merge: Less null warnings in src/ c78ff53
• Merge: Fix -e and -n on nit 4455de8
• Merge: nitc can inline redefinitions with -D d187822
• Merge: mnit: Fix error on refined virtual types 6064cf2
• Merge: Robust Virtual Type 10f8a25

Loader refactorization

The internal loader was refactorized and updated. The most important change is a simplification and an harmonization of the API. Especially, the class ModulePath was removed (in favor of using genuine MModule objects), and a scan_full method was added to analysis path passed as command arguments without loading the modules. By the way, the option -d (and -M) of nitls that were broken by the previous changes in the loader are now fixed, and nitcalatog works on directories of projects now. eg. nitcatalog ../lib

• Merge: Loader refactor 7743ff7
• Merge: loader: print error if bad files given to scan_full or parse_full e0c115e

Faster and more robust highlighting

This series fixes some long issues. The main change is the improvement of the method HightlightVisitor::enter_visit that highlights only the given node instead of the whole file.

Moreover, two new flags can be used to indicate how to deal with the context around the node to highlight.

• Merge: Improve highlight 96d13a4

Model services

The model, is extended with a basic visitor infrastructure. This offers an alternative for clients to navigate within a Nit model using a visitor design pattern. In the same time, is_fictive is promoted to any MEntity to denote internal entities that should not be exposed to the user.

• Merge: Introduce model_visitor 3c2c00e
• Merge: Generalize MModule::is_fictive to any entity a1863f2

Misc on tools

• Merge: parser: do not allocate a reduction table for each parser 3692191
• Merge: Detach CallSite from the AST a746049
• Merge: metrics: not a number fix d30d1c0
• Merge: metrics: add some Mendel metrics for classes 60a338c
• Merge: nitdoc: opts --no-render and --test f5dc56a

User documentation

We did a tedious work to detect and fix undocumented options in the manpages of tools. We also synchronized the description of --help and the text in manpages (when there is a diversion, the best one was often selected).

Little extra: a new script check_manpage can automatically check that new options of tools are documented in the man.

Moreover the README in the root directory was a little updated.

• Merge: Cleanup and synchronize options in manpages f8f0dcb
• Merge: Update readme d453adc

Automatic environment management

The script nit_env.sh tries to auto-magically configure PATH, MANPATH and bash completion for users. The point is to be as portable and simple as possible for the final user, as he just has to write

$ source misc/nit_env.sh

and get a working setup.

Moreover, if install is given as argument, then the script registers itself to the user's $HOME/.profile.

$ source misc/nit_env.sh install

One advantage is that the script invocation is registered in .profile, thus, future evolutions of the script will be used automatically in future sessions of the users.

• Merge: introduce nit_env.sh to setup the shell environment 96e6d45

Lib and contrib

Opportunity

Opportunity now stores local participant's name in a cookie. So we don't have to write our own name every single time we participate to an event.

It also features the ∇ ratio with a customizable multiplier. It assigns a weight to the choices of participants so that someone who select all possible answers puts less weight on each answer, and someone who chooses a single answer puts her full weight on it. It can be used to count the number of pizzas to order (with a multiplier of 0.4) and to give more importance to the available times of busy people when scheduling a meeting.

• Merge: contrib/opportunity: store local participant's name in a cookie d1342a4
• Merge: Opportunity: intro ∇ ratio 19f5ff8

Natural language processing

lib/nlp wraps Stanford CoreNLP and provides a set of natural language analysis tools which can take raw text input and give the base forms of words, their parts of speech, whether they are names of companies, people, etc., normalize dates, times, and numeric quantities, and mark up the structure of sentences in terms of phrases and word dependencies, indicate which noun phrases refer to the same entities, indicate sentiment, etc. This wrapper needs the Stanford CoreNLP jars that run on Java 1.8+.

var proc = new NLPProcessor("path/to/StanfordCoreNLP/jars")

var doc = proc.process("String to analyze")

for sentence in doc.sentences do
    for token in sentence.tokens do
            print "{token.lemma}: {token.pos}"
    end
end

• Merge: lib: Introduce Stanford CoreNLP wrapper 44e99da

Other notable features

• Merge: lib/graphs: introduce PageRank computation on digraphs a92d274
• Merge: lib/core: Added new hexdigest service on Text f0adfb2

Some optimizations

While working on nitlight, some performance improvements were made on the underlying libraries.

• Merge: lib/html: implement the tag list as an hashset instead of a array. 5273ee8
• Merge: More lib/html optimizations 3736fe8
• Merge: lib/core: Optimized html_escape for FlatText variants c3512b9
• Merge: Optimize hash collection 98eea9c
• Merge: Kill pseudo toplevel methods 4c2f2a3

Cleanup and buxfixes

Various cleanup and bugfixes

• Merge: Cleanup for Gamnit integration a662896
• Merge: FlatText is a private class 1c282b9
• Merge: Set HashMapIterator as a private class. 4e5486a
• Merge: lib/mongodb: minor fixes 2a68dff
• Merge: lib/github: clean code and add some services 801b722
• Merge: lib/serialization: fix deserialization constructor calling the default init 5ebe778
• Merge: remove some unwanted files 48c4d5d
• Merge: benitlux: produce better reports 60c2a08
• Merge: examples/shoot: fix typo in shoot_linux synopsys a2cdce3
• Merge: lib/trees: more services and documentation 5e73758
• Merge: Bench strings update b84161a