Progress Update

[Guillaume Cartier]

Hi everyone,

 

A lot has happened since the last message a month 1/2 ago. Let me try and summarize the most important developments :

 

Performance

=========

 

As I was saying in the last message, performance was very critical as this stage as it was so far behind the old Jazz as to make the new one unusable. The benchmark I have been using is editing Window.jazz a big file with lots of syntax coloring necessitating a lot of sexpr navigation. As every Lisp related tool must be able to navigate sexpr in text, those operations where are so critical that in the old Jazz they where handcoded directly in C. So keep in mind that in the benchmark results, it is actually handcoded C code vs Jazz code generating Gambit code.

 

First timing was :

 

Old Jazz : 0.3s

New Jazz : 28.5

 

so 95x slower.

 

We are now at 0.17s so 167x faster than the first version and 60% faster than the old handcoded C code. Note that there is still a lot of room to make this much faster but now that performance is really acceptable we are going to put performance aside for a while in favor of other priorities.

 

Thanks to Guillaume Germain for his wonderful statprof statistical profiler which really helped in finding the hot spots to optimize.

 

Type System

==========

 

Jazz now supports a fairly complete and completely optional type system.

 

Every part of a Scheme program can now be type annotated. Note that usually, just annotating the type of the input parameters of a function will be sufficient, as the new type system will use type inference whenever possible to deduce the type of expressions.

 

To support the new type system, a new experimental specializer / template system is also available. Any comments / suggestions on this part are most welcome as it is unclear what is the best approach: templates a la c++, generics a la java, parameterized types, ...

 

Note that Jazz as a whole doesn't use the new type system a lot. I'd say that around 30 files out of Jazz's 3000+ files use the new type system. Annotating only those 30 something critical modules resulted in huge performance improvements. Also note that the type annotations are purely optional and removing any of them will not change any code semantics.

 

IDE

===

 

The IDE while still missing a lot of functionality has become functional enough for it to become usable. I now try and use it as my main development IDE over the old Jazz's IDE.

 

It supports :

 

- Text editing

- Text dynamic re-evaluation

- Console evaluation

- Catalogs for navigation to symbol definitions and references

 

Mainly missing :

 

- Debugging support. We are still relying on the Gambit console. Note that this is really not painful at all as the Jazz generated is so close to the original code that reading a Gambit stack of Jazz code is almost identical as what a pure Jazz debugger stack would look like

 

Scheme Compliance

===============

 

The old Jazz, although "Scheme inspired" was never a full Scheme implementation.

 

Everything that Jazz was redefining from Scheme was rethought / renamed so that now I am very happy to report that Jazz is now a full R5RS Scheme (to be more precise: as R5RS as the underlying Scheme used) system where any R5RS Scheme code can be run as is inside a Jazz module or library.

 

If pure Scheme is used inside a Jazz module or library, the resulting code will have no overhead and will be the exact same code. The Jazz library system will just validate at compile time that every symbol referenced exists in the underlying Scheme system and report any unresolved symbols.

 

Multi-platform / X11

==============

 

The Jedi IDE while not functional, is now booting completely under X11! What was astonishing was that nothing was working, and a couple changes later, Jeremie had a version where *everything* was rendering perfectly! The port of the graphical code to Cairo really payed off there. I am very happy about this new development, especially as Jeremie was able to pull this off part time while studying for his master's degree. Gratz!

 

Git / Alpha

========

 

We now have a Git repo for JazzScheme.

 

At this point I am not sure we can even say that Jazz has reached alpha stage but if courageous people want to try it out, I feel that it is now stable enough to experiment with. If you are interested, just send me an email and I'll give you the address of the Git repo. The X11 port is not complete but any non-gui code should work on any Gambit supported platform.

 

Public beta release is still planned for very early 2008.

 

Changelog

========

 

At the present time as we are only 3 actively using the new Jazz, I am not keeping a detailed changelog. So, for those interested in more detail, here's a log of every Git commit since the Git repo was started 2 month ago :

 

2007-11-10 - Added profile-reset! to statprof + Various port fixes
2007-11-09 - New xlib event handlers and some fixes that make the IDE display correctly
2007-11-09 - Changed image load to relative path
2007-11-07 - Various function implementation that makes the event-loop run in X11
2007-11-06 - Some small formatting corrections
2007-11-06 - Added some X11 fonctions
2007-11-06 - Fixed windows dependency of some packages
2007-11-02 - Little bug fix with Display* in x11
2007-11-02 - Added X11 Key Pressed predicates
2007-11-02 - Flow-Layout fix
2007-11-09 - Cleanup of dev and doc directories
2007-11-09 - Put back the Numeric dynamic hierarchy + Many IDE port fixes
2007-11-08 - Removed all remaining functions that jazz duplicated from scheme
2007-11-08 - Removed the obsolete Map class
2007-11-08 - Renamed hashtable to table + Renamed many files to the new Jazz conventions
2007-11-08 - Modified make-table's parameters to be keyword based
2007-11-08 - Removed many obsolete classes
2007-11-07 - Removed the Printer class as Jazz now uses Scheme's output ports
2007-11-07 - Removed many functions that jazz duplicated from scheme
2007-11-07 - Put back the Sequence abstract class in order for sequences to be usable both dynamically and statically
2007-11-06 - Added a new Category-Type type that is now used to specialize new
2007-11-06 - Small scrolling optimization
2007-11-06 - Fixed the slow scrollbar scrolling that was due to rect-intersects-region? beeing unimplemented
2007-11-05 - Checkpoint commit before push
2007-11-05 - Converted String and Subseq to templates / specializers
2007-11-05 - Checkpoint in templates development
2007-11-03 - Converted Vector to specializers
2007-11-03 - Removed + * - / = < <= > >= from jazz.dialect.language
2007-11-02 - Removed the base class Numeric and the redifinitions of + and - in jazz.dialect.language
2007-11-02 - Converted Point and Dimension to specializers
2007-11-02 - Calls to virtual methods now correctly use the function type of the root method to determine call result type
2007-11-02 - Added a fixnum -> flonum cast to the language
2007-11-02 - Added parameter casts to methods + Converted Cell to specializers
2007-11-02 - Removed all public modifiers from slots as they do not make sense anymore
2007-11-02 - Added to specialize an option to specify the specializer name + Implemented specializer inlining
2007-11-02 - Implemented the specialize special form
2007-11-01 - Finished the macro / % special form unification for the walker and scheme walker
2007-11-01 - Removed the complex separation into macro / % special form from the code walker as it was not usefull anymore
2007-11-01 - Specializing new is now a special case
2007-11-01 - Slots now store their offset = object-size + rank for faster access
2007-11-01 - Special case optimisation for new with 0, 1 and 2 arguments + Object's initialize method now correctly takes 0 parameters
2007-11-01 - Base object size is now included in instance size for faster object creation
2007-11-01 - Added ##table-merge! to Gambit
2007-11-01 - Fixed a small bug in jscm compilation + Removed Reader classes in favor of Scheme ports
2007-11-01 - Removed the obsolete #\eof from the language
2007-11-01 - Jazz preexpanded .jscm files and now generated in the bin directory
2007-11-01 - Now reuses the already read macro list in walk-library when parsing the library declarations
2007-11-01 - Various performance improvements to load time
2007-11-01 - Optimized walker to walk a library's declarations only once + Cleanup of the autoloads catch all jazz.library
2007-10-31 - Statprof now walks the continuation chain backward to try and find a continuation with source info + Statprof can now generate an sexp report that Jedi can view + Optimized char test methods + Methods now support inner defines + Refactored Windows common dialogs code
2007-10-31 - Method dispatch now uses %%get-object-class for non-primitive classes
2007-10-31 - Small improvement for statprof to sort results + define-virtual-syntax now uses %%get-object-class
2007-10-30 - Removed some user level tests from generated code in release
2007-10-27 - Added all .jazz files as I forgot to do the add in the previous commit
2007-10-27 - Renamed every .fusion file to .jazz
2007-10-27 - Various port / type system fixes
2007-10-26 - Removed false and true constants from the language
2007-10-26 - Added inlining support for definitions, final methods and dispatch to final methods
2007-10-25 - Fixed many catalog related port bugs
2007-10-25 - Added specialization based on static parameter types
2007-10-24 - Fixed a bug in restricted annotations propagation
2007-10-23 - Work on the X11 platform
2007-10-23 - Implemented conditional branch type inference
2007-10-22 - Added optional, named and rest parameter support to the type system and many other features
2007-10-20 - Implemented efficient %%class-subtype? and %%class-is? algorithms
2007-10-20 - Inlined dispatchers inside libraries beeing compiled
2007-10-20 - Optimized direct method access and implemented dispatch caching
2007-10-19 - Completed the vtable based virtual methods + Added a validation for modules and libraries to ensure their name is consistant with their pathname
2007-10-19 - Declarations are now types to support compile time type analysis + Moved Gambit's headers that we use into our include directory
2007-10-18 - Implemented specifier parsing for name, function and template types
2007-10-18 - Converted all generics in language to methods
2007-10-18 - Converted fold-literal to a method
2007-10-17 - Added vtable dispatch to class and interface methods (Stephane Le Cornec)
2007-10-16 - Many improvements to the type inference system
2007-10-15 - Added type checks on every declared types
2007-10-15 - Renamed the void concept to unspecified in preparation to having a real void type
2007-10-14 - Added cond-expand to the language
2007-10-14 - Cleanup of windows specific related concepts (Jeremie)
2007-10-14 - Checkpoint commit before merge
2007-10-14 - All emit primitives now recursively return a code object containing the generated code and it's type
2007-10-14 - Converted all emit code to take an annotated environment parameter
2007-10-13 - Major cleanup of everything related to signatures that are now full fledged objects
2007-10-12 - Changed safety-level to a cleaner more consistent Scheme feature
2007-10-12 - Converted all text files to Unix line endings
2007-10-11 - Fixed some bugs due to the new directory structure
2007-10-11 - Major move around of files based on the concept of launching gsc from the bin/buildprefix directory
2007-10-10 - Removed the direct slot access syntax [] from the language and converted all the packages
2007-10-09 - Added support for direct self slot access through the slotname~self notation
2007-10-09 - Moved some files to legacy directory
2007-10-09 - Checkpoint commit before big search & replace
2007-10-08 - References to autoload declarations now use only once generated autoload functions
2007-10-08 - Moved statprof to the new contrib directory
2007-10-08 - Cleanup and better naming of many root files
2007-10-08 - Converted all calls to inneficient list? scheme primitive + Made is? a core virtual method based on the type
2007-10-05 - Merge with Jeremie
2007-10-05 - Checkpoint commit before merge
2007-10-05 - Added a new Type class that is the superclass of Category
2007-10-05 - Renamed unit with a more appropriate naming: category
2007-10-05 - Various fixes in the code walker
2007-10-05 - Merge branch 'walker'
2007-10-05 - Completed walker redesign where a full expression parse tree is first built and then emited to Scheme code
2007-10-01 - Checkpoint commit to switch branch
2007-10-01 - Added catalog/jazz.dev.jcat to source control
2007-09-29 - Fixed a small bug
2007-09-29 - Added the catalog to git to simplify switching branch
2007-09-29 - Checkpoint in walker redesign
2007-09-28 - Renamed types like <String> to <string> to reflect their primitive status in Scheme
2007-09-28 - Added %% to some missing places in kernel code
2007-09-28 - Removed all enumerator / keyword equivalence tests
2007-09-28 - Completed migration of the code to srfi-89 optional and named parameters
2007-09-27 - Check point in the marathon on converting all the code to Scheme style keywords
2007-09-27 - Made generic and specific aware of optional and named parameters
2007-09-27 - Modified generic and specific syntax to require type specifiers to make them distinguishable from optional parameters
2007-09-27 - Added support for optional and named parameters in lambda signatures + Converted some code
2007-09-26 - Added support for signature named let and converted all code to it
2007-09-26 - Replaced many call/ec iterations in language with named let iteration + Renamed fixnum macros by fx
2007-09-26 - Optimized slot access + Added library variables concept to support slot access optimisation
2007-09-26 - Switched Jazz's delimited string syntax from #||# to #"#
2007-09-26 - Added preliminary support for Gambit structure based types + Various port fixes
2007-09-25 - Fixed bugs due to make-string's default char beeing #\nul + Fixed (cdr (ass*)) bugs
2007-09-25 - Standardize text line separator in strings to #\newline + Port fixes
2007-09-25 - Many port fixes + Made console and text evaluate work with the underlying Scheme
2007-09-25 - Various port fixes + Added conditional features to modules and libraries
2007-09-24 - Finished removing the null constant from the language
2007-09-24 - Replaced alot of null by either {} or '()
2007-09-23 - Fixed a bug in hiword + Put back the calls to DeleteObject
2007-09-22 - Jeremie's memory leak fixes
2007-09-22 - Completed apply-diff + Various fixes
2007-09-21 - Unified Jazz's car and cdr semantics with Scheme's
2007-09-21 - Removed the nil constant from the language in favor of the {} literal
2007-09-21 - Removed nil? and not-nil? from the language
2007-09-21 - Modified the Jazz language and the library to have the same semantics as Scheme for what is considered true and false
2007-09-21 - Removed obsolete Fusion-* classes + Text explorer now functional
2007-09-20 - Various port fixes + Fixed a bug where generated literal names could clash under separate compilation
2007-09-20 - Merge with Stephane
2007-09-20 - UI big fixes + Added a temporary user library for tests
2007-09-19 - Fixed message-box and other port related bugs
2007-09-19 - Removed obsolete OLE stuff
2007-09-19 - Fixed offscreens and tree related bugs
2007-09-19 - Various fixes
2007-09-19 - Renamed products -> packages
2007-09-18 - Various migration fixes +
2007-09-17 - Jeremie's modifications. Cursors and various Cairo fixes
2007-09-15 - Cleanup of numerics. Removed Integer and Real classes and added Numeric
2007-09-15 - Finished %%c-class-of implementation + cleaned walker references + started work on cffi
2007-09-14 - Migrated jazz.platform.external to the Jazz dialect walker
2007-09-14 - Various fixes
2007-09-14 - Added jazz.walk-for parameter and implemented %%class-of using ##c-code
2007-09-14 - Changed meroon subtype for the new jazz subtype :)
2007-09-14 - Various fixes and completed converting the module macro to ##define-syntax
2007-09-13 - Merge branch 'master' of ssh://jaz...@trex.iro.umontreal.ca/~jazzscm/jazz
2007-09-13 - Various fixes for docked-root frame drawing.
2007-09-13 - Removed .jcat files from git repository
2007-09-13 - Merge with Jeremie
2007-09-13 - Work in progress before merge
2007-09-12 - Many fixes
2007-09-11 - Various fixes
2007-09-11 - Various fixes for the Gambit port
2007-09-10 - Some text fixes and added the catalog to the distribution
2007-09-09 - Various fixes for Jedi console display.
2007-09-10 - Work in progress: experimentation in class-of optimization and define-syntax usage
2007-09-08 - Class and interface access optimization by hashtables
2007-09-08 - Library access optimization by hashtables
2007-09-06 - Merge with Jeremie
2007-09-06 - Work in progress
2007-09-05 - Core implementation of vtables + cleanup + statprof
2007-09-04 - Initial commit

Guillaume

[Guillaume Cartier]

Yes, I reread the Install.txt and it is not well explained!

 

The problem you are having is that while the Jazz core (e.g. everything not GUI) car run 100% Gambit interpreted, the GUI modules use Gambit's FFI and so need to be compiled. This you see from Gambit's "Interpreter does not support ##c-define-type" message.

 

To fix this, just execute first (ball) that will build the jazz core + language + platform modules. Quit gsc. Relaunch gsc and then execute (j). I suggest you use the GambitWinIntelDebug folder and also if everything works correctly to launch the IDE, try running (bjd) : it will compile all IDE critical files and make it *alot* faster.

 

PS: I'm cc'ing the group in case others have the same problem.

Guillaume

On Nov 15, 2007 8:09 PM, Deepankar Sharma <deepanka...@gmail.com> wrote:

Hello Guillaume,

I get an error trying to boot up Jazzscheme (this is on windows ). I
have gambit 4.1 and the sources for it installed as specified,
jazzscheme is pulled from the git repo approximately at 8:00 pm US
Eastern time.

-------------------------------------------------------------------------------------------------
*** ERROR IN "..\\..\\packages\\org.jazz\\lib\\jazz\\platform\\cairo\\_cairo.jaz
z"@38.1 -- Interpreter does not support ##c-define-type
(current-exception-handler) = primordial-exception-handler
(current-input-port) = '#<input-output-port #2 (console)>
(current-output-port) = '#<input-output-port #2 (console)>
(current-directory) = "C:\\Users\\deep\\Desktop\\jazzscheme\\dev\\bin\\Gambi...
(current-readtable) = '#<readtable #3>
(jazz.walk-for ) = 'interpret
(jazz.requested-module-name) = 'jazz.platform.cairo
(jazz.load-indent) = 8
(##compilation-scope) = '#(())
1>
--------------------------------------------------------------------------------------------------

My main dev platform is linux/sbcl but I am trying this on windows
since I want to give the IDE a go. Once things start working, I do
want to start tinkering with scheme.

Regards,
Deepankar

[Christopher Diggins]

On Nov 15, 2007 9:09 PM, Guillaume Cartier <gcar...@jazzscheme.org> wrote:
> Yes, I reread the Install.txt and it is not well explained!
>
> The problem you are having is that while the Jazz core (e.g. everything not
> GUI) car run 100% Gambit interpreted, the GUI modules use Gambit's FFI and
> so need to be compiled. This you see from Gambit's "Interpreter does not
> support ##c-define-type" message.
>
> To fix this, just execute first (ball) that will build the jazz core +
> language + platform modules. Quit gsc. Relaunch gsc and then execute (j). I
> suggest you use the GambitWinIntelDebug folder and also if everything works
> correctly to launch the IDE, try running (bjd) : it will compile all IDE
> critical files and make it *alot* faster.

Any chance of putting this (and other such installation things) in an
install script? That would be very convenient.

- Christopher

[Guillaume Cartier]

This is really an install script but written in Scheme. When you are executing (ball) which I should rename to just (build), you are using Gambit as a script engine to execute the script that will build Jazz. When the script is done building, you quit the script engine: Gambit. Then you launch Gambit again to run the newly built Jazz.

 

Creating multiplatform scripts can be difficult and as Gambit abstracts all those concerns into a high level API, Jazz follows its philosophy of trying to do everything as possible in Lisp.

 

Let me try and explain a bit more. At the current time, Jazz is actually the following tower of languages :

• Scheme / Gambit
• kernel : minimal scheme code that adds modules that can be required
• class : module that implements low-level classes
• generic : module that implements generic functions
• core : module that implements libraries
• scheme : library that implements the Scheme language
• jazz : library that implements the Jazz language

Great care was taken in order for all those modules / libraries to be able to run 100% interpreted in the underlying Scheme. Beeing able to run something as complex as a programming language 100% interpreted has played (and still plays) a big part in how easy and fast it has been to develop Jazz in Gambit.

Also, Gambit has a very nice feature where interpreted and compiled code can be freely mixed, so really it is possible to modify the Jazz language anywhere with 0 rebuilding. The file you modified will just now be loaded interpreted. So basically when you run (build), you are just optimizing the speed of the core modules.

 

Of course this is all interesting only to developers interested in understanding the Jazz language itself and I plan on having binary distributions for the standard platforms Mac and Windows. With a binary distribution, running the IDE will simply be launching the  Jedi.exe application.

 

Hope this helps,

 

Guillaume