Some beginner's questions

[Johannes Engels]

Hello,

as a geodesist with some basic knowledge in ML, I recently stumbled
across Qi and got immediately attracted. The Qi book is my companion
in the commuter train now, and I am excited about Qi, mostly about the
mere beauty of its type system. However, so far it is more the kind of
beauty of an exhibit behind a showcase for me, as the Qi book seems to
lack some information which would make the language usable in real
life. So I should like to bother you with the following beginner's
questions:

1) Is there any support for arrays? Whereas lists, trees etc. are
useful of course, for me by far the most important data structure is
the array. How can I read an array from file? (Working in image
processing, I need to read matrices from files hundred times per day).
Are there library functions available for this task? Is there a
function comparable to Haskell's "listArray", which converts lists
into arrays? Is there a binding to GSL or Lapack or a comparable
library for linear algebra? Am I supposed to write such functions
myself? Or should I better learn Common Lisp and use CL's library
functions? Are Qi arrays and CL arrays the same or do I have to
convert between them?

2) Is there a module system in Qi comparable to ML's module system? Or
how are bigger Qi programs structured?

3) Are there different modes for compiling or interpreting a program
as in Haskell / Ocaml ? In Appendix B of the Qi book there is a
paragraph about "Communicating with the Merlin Compiler", but here I
am completely lost.

4) A minor question: Is there a possibility to show previous
expressions in the toplevel without immediate evaluation? % shows
previous inputs, but does not give the opportunity to EDIT them and
then submit anew. (I am working under Linux.)

5) Do you recommend to use Qi for scientific programming at all ?
(Performance does not matter much to me.) Or is it intended more as a
testbed for computer scientists?

Any hint is welcome!
Best regards
Johannes Engels

[Mark Tarver]

Qi is now being supplanted by Shen which is a portable version of Qi
so far running under Javascript and Common Lisp. Shen is however very
similar to Qi (type system etc) and nearly everything that runs under
Qi runs under Shen. I'll answer your questions wrt Shen.

To see the differences between Shen and Qi you can read this:

http://www.shenlanguage.org/Documentation/shendoc.htm

On Mar 6, 9:28 am, Johannes Engels <johannes.eng...@hft-stuttgart.de>
wrote:

> Hello,
>
> as a geodesist with some basic knowledge in ML, I recently stumbled
> across Qi and got immediately attracted. The Qi book is my companion
> in the commuter train now, and I am excited about Qi, mostly about the
> mere beauty of its type system. However, so far it is more the kind of
> beauty of an exhibit behind a showcase for me, as the Qi book seems to
> lack some information which would make the language usable in real
> life. So I should like to bother you with the following beginner's
> questions:
>
> 1) Is there any support for arrays? Whereas lists, trees etc. are
> useful of course, for me by far the most important data structure is
> the array. How can I read an array from file? (Working in image
> processing, I need to read matrices from files hundred times per day).
> Are there library functions available for this task? Is there a
> function comparable to Haskell's "listArray", which converts lists
> into arrays? Is there a binding to GSL or Lapack or a comparable
> library for linear algebra? Am I supposed to write such functions
> myself? Or should I better learn Common Lisp and use CL's library
> functions? Are Qi arrays and CL arrays the same or do I have to
> convert between them?

Shen uses vectors and these can of course have vector elements
themselves. The vectors are adapted from the native platform.

http://www.shenlanguage.org/Documentation/shendoc.htm#Vectors

There is support for reading bytes (from image files too)

http://www.shenlanguage.org/Documentation/shendoc.htm#Streams

If you read these destructively into a vector should be fairly fast.

>
> 2) Is there a module system in Qi comparable to ML's module system? Or
> how are bigger Qi programs structured?

There is a package system which is part of the language. I think (may
be wrong) that there is a module system in github but we have not
moved this to the standard library.

http://www.shenlanguage.org/Documentation/shendoc.htm#Packages

>
> 3) Are there different modes for compiling or interpreting a program
> as in Haskell / Ocaml ? In Appendix B of the Qi book there is a
> paragraph about "Communicating with the Merlin Compiler", but here I
> am completely lost.

Qi had a lot of optimisations for running under CL. Shen is simpler;
you don't have to do anything like that.

>
> 4) A minor question: Is there a possibility to show previous
> expressions in the toplevel without immediate evaluation? % shows
> previous inputs, but does not give the opportunity to EDIT them and
> then submit anew. (I am working under Linux.)

Nothing like that. but you could ask the people who use SLIME and
Emacs because I expect that you can do that sort of thing.

> 5) Do you recommend to use Qi for scientific programming at all ?
> (Performance does not matter much to me.) Or is it intended more as a
> testbed for computer scientists?

Yes; Shen is going to have its maths library this month and I think
you could well run it for those purposes. The integration with CL in
Shen is as good as Qi and that means you can integrate all the
scientific software written under CL.

Mark

[SteveN]

Greetings,

Given the title of this thread, it might might be useful to put all the newbie questions here, such as this one.

There's reference to libraries; two of them now, maths and strings, however I can't find mention of how to use them? For example I need to use the math lib function (round), but shen claims that it's not defined. How does one load the libraries into the environment? My first guess would be (load "some-file.shen", but I can't even find the libraries. Are they somewhere on the shenlanguage.org website? The strings library is there: http://www.shenlanguage.org/library.html, but not maths.

Cheers,

    - SteveN

[Mark Tarver]

Willi has been working on the maths, and the code is there but we need
to get this up. We also have a vector library too, but that is beta.
However again I want to push ahead on this one. We'll see if we can
get it all up within a week. I agree that talking about a 'round'
function that nobody can use is not a solution.

Mark

> > Johannes Engels- Hide quoted text -
>
> - Show quoted text -

[Johannes Engels]

On Mar 6, 12:18 pm, Mark Tarver <dr.mtar...@gmail.com> wrote:
> Qi is now being supplanted by Shen which is a portable version of Qi
> so far running under Javascript and Common Lisp.  Shen is however very
> similar to Qi (type system etc) and nearly everything that runs under
> Qi runs under Shen.  I'll answer your questions wrt Shen.
>

Thank you for your kind and detailed reply! As I was busy last weeks,
I can continue only now.
Beforehand I prefer to stick with Qi for several reasons:

- Portability is not that important for me
- Qi is better documented (I have the Qi book, who knows when the Shen
book will be available)
- Qi seems to be more mature compared to Shen. Only one example:
in Qi:
(0-) (+ 0.1e-02 0.2e-02)
0.003

in Shen:
(0-) (+ 0.1e-02 0.2e-02)
+: e-02 is not a number

This functionality is indispensable for me, but I don't want to
reinvent the wheel myself.
- Qi seems to have better support of arrays; in particular it supports
multidimensional arrays,
whereas in Shen there are only arrays of arrays.

> Shen uses vectors and these can of course have vector elements
> themselves. The vectors are adapted from the native platform.
>
> http://www.shenlanguage.org/Documentation/shendoc.htm#Vectors
>
> There is support for reading bytes (from image files too)
>
> http://www.shenlanguage.org/Documentation/shendoc.htm#Streams
>
> If you read these destructively into a vector should be fairly fast.

I would appreciate very much if there were library functions for
reading matrices / images from files! Here comes my (rather crude)
effort to read a matrix from an ASCII-file; I would appreciate if
somebody could comment on this although it is Qi and not Shen:

(define matrix-from-file
NumRows NumCols File ->
(let Content (read-file File)
(list-to-matrix Content NumRows NumCols)
)
)

(define list-to-matrix
MyList NumRows NumCols ->
(if (= (length MyList) (* NumRows NumCols) )
(put-loop (make-array [NumRows NumCols] )
NumRows
NumCols
0
0
MyList
)
(error "Wrong matrix dimensions ~%")
)
)

(define put-loop
Mx NumRows NumCols Row Col MyList ->
(if (>= Row NumRows)
Mx
(if (>= Col NumCols)
(put-loop Mx NumRows NumCols (+ Row 1) 0 MyList)
(let Dummy (put-array Mx [Row Col] (head MyList) )
(put-loop Mx
NumRows
NumCols
Row
(+ Col 1)
(tail MyList)
)
)
)
)
)

(define read-matrix-show-element
File NumRows NumCols Row Col ->
(let Mx (matrix-from-file NumRows NumCols File)
(if (and (< Row NumRows) (< Col NumCols) )
(get-array Mx [Row Col] unknown)
(error "Index out of range ~%")
)
)
)

The last function is only for testing of course. Having a file
"testdata" containing the numbers "1 2 3 4 5 6" I get
(1-) (read-matrix-show-element "testdata" 2 3 1 0)
4
So this seems to work so far.
Opposite to the book (p. 99) function "make-array" seems to take the
dimensions themselves as input, not (dimensions - 1).
The construction "let Dummy (...)" in the function "put-loop" looks
ugly, but so far I could not figure out if there is a possibility to
chain impure functions (which do the destructive updates) disregarding
their results. Does such a possibility exist?

Best regards
Johannes

PS: I can well imagine that for computer scientists this seems all
boring and unimportant, but for engineers it is not! In Matlab, there
is "load filename", in numpy / Python there is "genfromtxt" and
"reshape" etc ... I guess I could not convince any colleague of mine
to try a programming language, however well conceived, which does not
offer this functionality!

[Mark Tarver]

On Mar 29, 1:20 pm, Johannes Engels <johannes.eng...@hft-stuttgart.de>
wrote:

> On Mar 6, 12:18 pm, Mark Tarver <dr.mtar...@gmail.com> wrote:
>
> > Qi is now being supplanted by Shen which is a portable version of Qi
> > so far running under Javascript and Common Lisp.  Shen is however very
> > similar to Qi (type system etc) and nearly everything that runs under
> > Qi runs under Shen.  I'll answer your questions wrt Shen.
>
> Thank you for your kind and detailed reply! As I was busy last weeks,
> I can continue only now.
> Beforehand I prefer to stick with Qi for several reasons:
>
> - Portability is not that important for me
> - Qi is better documented (I have the Qi book, who knows when the Shen
> book will be available)
> - Qi seems to be more mature compared to Shen. Only one example:
>       in Qi:
>       (0-) (+ 0.1e-02 0.2e-02)
>       0.003
>
>       in Shen:
>       (0-) (+ 0.1e-02 0.2e-02)
>       +: e-02 is not a number
>

I could not reproduce that behaviour I get

Shen 2010, copyright (C) 2010 Mark Tarver
www.lambdassociates.org, version 3.0
running under Common Lisp, implementation: CLisp 2.49
port 1.1 ported by Mark Tarver

(0-) (+ 0.1e-02 0.2e-02)
0.003

(1-)

>       This functionality is indispensable for me, but I don't want to
> reinvent the wheel myself.
> - Qi seems to have better support of arrays; in particular it supports
> multidimensional arrays,
>   whereas in Shen there are only arrays of arrays.

I'm not certain that a two dimensional array and a vector of vectors
of uniform length have that much to seperate them.

>
> > Shen uses vectors and these can of course have vector elements
> > themselves.  The vectors are adapted from the native platform.
>
> >http://www.shenlanguage.org/Documentation/shendoc.htm#Vectors
>
> > There is support for reading bytes (from image files too)
>
> >http://www.shenlanguage.org/Documentation/shendoc.htm#Streams
>
> > If you read these destructively into a vector should be fairly fast.
>
> I would appreciate very much if there were library functions for
> reading matrices / images from files! Here comes my (rather crude)
> effort to read a matrix from an ASCII-file; I would appreciate if
> somebody could comment on this although it is Qi and not Shen:

You'll have problems in Qi; it was not designed to read files as byte
level though Shen is.

Mark

[Johannes Engels]

On Mar 29, 3:22 pm, Mark Tarver <dr.mtar...@gmail.com> wrote:

> I could not reproduce that behaviour I get
>

> Shen 2010, copyright (C) 2010 Mark Tarverwww.lambdassociates.org, version 3.0

> running under Common Lisp, implementation: CLisp 2.49
> port 1.1 ported by Mark Tarver
>
> (0-) (+ 0.1e-02 0.2e-02)
> 0.003
>
> (1-)
>

I just reinstalled Shen (under Ubuntu Linux 11.10); however the
behaviour was as in my previous post. Does somebody have suggestions?

Best regards
Johannes

[Johannes Engels]

On Mar 29, 3:22 pm, Mark Tarver <dr.mtar...@gmail.com> wrote:
>
> I'm not certain that a two dimensional array and a vector of vectors
> of uniform length have that much to seperate them.
>

You are certainly right. However, let us assume we wish to retrieve an
element T(i1, i2, i3, i4) of a tensor T of fourth order (e.g. Hooke's
tensor), I guess we would have to write something like

(<-vector (<-vector (<-vector (<-vector T i1) i2) i3) i4),

and obviously some syntactic sugar or a wrapper function would be
convenient. Or would you prefer to store the elements of a
multidimensional array in a one-dimensional vector and to map the
multidimensional index i_k to a one-dimensional index via

running index = sum_k (i_k * product_(j=k+1) r_j ),

where r_j are the respective index ranges? I mean, we could define
something like

(datatype multidim-array

if (= Dimension (length IndexRanges))
IndexRanges : [integer];
Content : (vector number);
=====================================
[Dimension IndexRanges Content] : multidim-array;
)

and define getter- and setter functions. However, I have no idea

a) what this would mean for performance and
b) if this would impede interaction with any Common Lisp libraries,
e.g. GSLL.

Do you have any suggestions?

Best regards
Johannes

[Mark Tarver]

If you want to access an n-dimensional array and want to avoid nesting
<-vector, then you could define a recursive function that took an
array and an integer list and recursed. If you wanted both speed and
type security, you probably want a macro that takes an expression like

(<-array Array 1 2 3 4)

and expands it to a series of vector calls. You need to couple it
with something like a type (array A N) where N is a natural number > 0
which is macroexpanded to a nested series of vector types (vector
(vector .... A)))).

Mark

On Mar 30, 9:06 am, Johannes Engels <johannes.eng...@hft-stuttgart.de>
wrote:

[Joel Shellman]

Regarding multidim arrays... I realize I'm brand new here, but I'll
suggest a couple things just in case it might be useful. My
understanding is that there are important differences between a vector
of vectors and a multidim array. For example, a multidim array should
enforce the size of all columns and rows to be the same.

My first guess would be to implement a multidim array as a single
array with the dimensions mapped onto that.

I have a question about what Shen can do regarding types related to this...

Can I include the size of an array or multidim array in the type? Then
declare something to be of type: [two-dim-array 4 5]. And then define
a matrix multiplication function something like this:
{(two-dim-array n m) --> (two-dim-array m p) --> (two-dim-array n p)}

Is something like this encoding the size of matrices in the type
system possible in Shen?

[Mark Tarver]

Short answer; yes. Longer answer; it leads into deeper waters than
you need to be in as a beginner. Best advice is to learn macros and
follow the course advised. You can easily create a function that
allows you to create non-jagged arrays.

Mark

[Joel Shellman]

> Short answer; yes.  Longer answer; it leads into deeper waters than
> you need to be in as a beginner.

I'm a beginner in Shen, but I've been around the block a few times in
other languages. These are the things I need to learn in order to do
what I want with the language.

It is said: "Shen's type system is more powerful than that any
existing functional language including ML and Haskell." What about
type systems in things like Agda? Ah, this looks great:
http://sandycat.info/blog/deptypes-shen/
There's the answer to the question about matrix dims in the type right
there in that blog post. Awesome!
(and that's something about performance possibilities--with that type
of thing you don't need bounds checking on safevec accesses)

> Best advice is to learn macros and follow the course advised.

"course advised"? what course? Do you just mean reading all the Shen
documentation?

Thanks!

[Mark Tarver]

The comparison with Agda etc. has come up before. Its difficult to
compare systems unless you have used both intensively but.here is my
take.

First I think it is probably a bit misleading to think of Shen as
defined by a type system because it supplies primarily a notation for
defining types. That notation is in fact a programming language in
its own right and that really is the basis for the 'most powerful'
claim. ?逶蠠љunless you are only specific

The actual type system inside Shen is simple but powerful and directed
to largely automatic type checking. Systems like e.g. NuPrL etc. are
designed to deal with more complex types but require user interaction
and programming of tactics. Such work is possible in Shen, but it
belongs to the book after the one I am writing.

I also query whether some of these systems are practical programming
**languages** rather than tools for experimenting with types or proof
assistants. I feel that one mark of maturity is that a language
should be powerful enough to bootstrap itself. I don't think anybody
has tried or will try e.g. synthesising NuPrl in constructive type
theory whereas Shen is 98% Shen and almost entirely bootstrapped.

Hence an attempt to compare these systems is rather like comparing a
racing car to an earth digger; it rather depends on what you are
trying to do as to which one seems to you 'better' which means that
'better' of itself is probably not meaningful here.

Mark

[Mark Tarver]

> "course advised"? what course? Do you just mean reading all the Shen
> documentation?

This is to start you off; ok?

(package my-array-stuff- [<-array make-array]

(defmacro <-array-macro
[<-array A | Ns] -> (<-array-macro-help A (reverse Ns)))

(defmacro array-macro
[array N A] -> (array-macro-help N A))

(define array-macro-help
1 A -> [vector A]
N A -> [vector (array-macro-help (- N 1) A)])

(define <-array-macro-help
A [N] -> [<-vector A N]
A [N | Ns] -> [<-vector (<-array-macro-help A Ns) N])

(defmacro make-array-macro
[make-array | Ns] -> (make-array-macro-help (reverse Ns)))

(define make-array-macro-help
[N] -> [vector N]
[N | Ns] -> [fill-vector (make-array-macro-help Ns) N])

(define fill-vector
Vector N -> (fill-vector-help Vector 1 (limit Vector) N))

(define fill-vector-help
Vector Limit Limit N -> (vector-> Vector Limit (vector N))
Vector Count Limit N -> (fill-vector-help (vector-> Vector Count
(vector N)) (+ 1 Count) Limit N)) )

Examples:

(2-) (make-array 2 3)
<<... ... ...> <... ... ...>>

(3-) (<-array (@v (@v a b <>) (@v c d <>)) 1 2)
b

Mark

[Johannes Engels]

Let me please ask some more questions regarding the Qi syntax:

1) On page 53 of the Qi book there is a statement about wildcards:
"if a variable occurs on the left-hand side of the -> only once, and
it does not occur anywhere on the right-hand side of the ->, then it
may be replaced by a wildcard."
On page 187, however, there is a function "parameters" defined by

(define parameters
[] -> []
[_ | Attributes] -> [_ | (parameters Attributes)])

so the wildcard is appearing on both sides of the ->. How is this
evaluated? Still I do not understand what the purpose of the function
"parameters" is.

2) Still it is not clear for me how currying works in Qi. If I write

(0-) (map (+ 3) [4 5 6])
[7 8 9]

it works as expected; I don't need a lambda such as
(map (/. A (+ 3 A)) [4 5 6]).
But on page 186 (bottom) there is a function definition

(define selectors
Name Attributes -> (map (/. A (selector Name A)) Attributes))

and indeed, if I replace this by

(define selectors
Name Attributes -> (map (selector Name) Attributes))

I get a runtime error. What are the rules here?


3) If I switch type-checking on and load the file "Chap12/
structures(typed).qi", I get a compilation error:
"error: defstruct has no signature", although there is after the
function definition a snippet
(declare defstruct ...).
Am I supposed to load this file with typechecking switched off? But
why is then the "declare" function needed?

4) In file "Chap12/structures(untyped).qi" there is a function
"assoc" applied, which seems to be defined neither in
the file nor in the list of system functions. (The file compiles
anyway.) What does "assoc" do? Is there a possibility to list all the
bound variables and function names in the interpreter ?

I would be grateful for any hints - sorry if I overlooked some
explanations in the book!

Best regards
Johannes

[Willi Riha]

Mark is rather busy at the moment, so I will try to answer your question:

a wild card is essentially a placeholder, and is used where a value does
not matter.
In function parameters, the head of the list is obviously not important,
and is represented
by a wildcard on both sides of the ->. Exactly what this function does
I cannot tell you
without studying section 12.2 of the book.

Willi

[Mark Tarver]

On Apr 20, 4:53 pm, Johannes Engels <johannes.eng...@hft-stuttgart.de>
wrote:

> Let me please ask some more questions regarding the Qi syntax:
>
> 1) On page 53 of the Qi book there is a statement about wildcards:
> "if a variable occurs on the left-hand side of the -> only once, and
> it does not occur anywhere on the right-hand side of the ->, then it
> may be replaced by a wildcard."
> On page 187, however, there is a function "parameters" defined by
>
> (define parameters
>    [] -> []
>    [_ | Attributes] -> [_ | (parameters Attributes)])
>
> so the wildcard is appearing on both sides of the ->. How is this
> evaluated? Still I do not understand what the purpose of the function
> "parameters" is.

In Qi, not Shen, a _ to the RHS created an arbitrary variable (just
like the LHS).

>
> 2) Still it is not clear for me how currying works in Qi. If I write
>
> (0-) (map (+ 3) [4 5 6])
> [7 8 9]
>
> it works as expected; I don't need a lambda such as
> (map (/. A (+ 3 A)) [4 5 6]).
> But on page 186 (bottom) there is a function definition
>
> (define selectors
>     Name Attributes -> (map (/. A (selector Name A)) Attributes))
>
> and indeed, if I replace this by
>
> (define selectors
>     Name Attributes -> (map (selector Name) Attributes))
>
> I get a runtime error. What are the rules here?

If you entered 'selectors' first Qi would not know that a partial
application was called for in the second case and would not make one.

>
> 3) If I switch type-checking on and load the file "Chap12/
> structures(typed).qi", I get a compilation error:
> "error: defstruct has no signature", although there is after the
> function definition a snippet
> (declare defstruct ...).
> Am I supposed to load this file with typechecking switched off? But
> why is then the "declare" function needed?

From memory 'yes'; the program is not type checked but the operation
of it is.

>
> 4) In  file "Chap12/structures(untyped).qi" there is a function
> "assoc" applied, which seems to be defined neither in
> the file nor in the list of system functions. (The file compiles
> anyway.) What does "assoc" do? Is there a possibility to list all the
> bound variables and function names in the interpreter ?

See appendix A.

>
> I would be grateful for any hints - sorry if I overlooked some
> explanations in the book!
>
> Best regards
> Johannes

You need to get away from Qi, because we're all moving to Shen.

Mark

[Johannes Engels]

Afaics this solution is only correct for arrays up to dimension 2 (try
e.g. (make-array 2 3 4) ). The snag seems to be in the third line of
make-array-macro-help: Whatever multi-dimensional nested array the
expression (make-array-macro-help Ns) is returning, the function fill-
vector replaces each element of the *outermost* vector by a vector of
length N, so that at most a 2D-array is obtained. I guess the
following solution is more general:

(define nestled-array
[N] Init -> (let New-Vector (vector N)
(vec-init New-Vector Init))
[N | Ns] Init -> (let New-Vector (vector N)
(nest-array-set-elems New-Vector 1 N Ns
Init)
)
)

(define nest-array-set-elems
Vector Limit Limit Ns Init ->
(vector-> Vector Limit (nestled-array Ns Init))
Vector Index Limit Ns Init ->
(let New-Vec (vector-> Vector Index (nestled-array Ns Init))
(nest-array-set-elems New-Vec (+ Index 1) Limit Ns
Init)
)
)

(define vec-init
Vector X -> (let Limit (limit Vector)
(vec-init-rec Vector 1 Limit X)
)
)

(define vec-init-rec
Vector Limit Limit X -> (vector-> Vector Limit X)
Vector Index Limit X ->
(vec-init-rec (vector-> Vector Index X) (+ Index 1) Limit
X)
)


Example:
(2-) (nestled-array [2 3 4] 0.0)
<<<0 0 0 0> <0 0 0 0> <0 0 0 0>> <<0 0 0 0> <0 0 0 0> <0 0 0 0>>>

Did not use macros so far. Without initialization all elements would
have the value "fail!" (is this correct?).


This was a bit nit-picking so far. More important for me is the
question: Are these nested vectors compatible with Common Lisp arrays?
I would like to use a package for linear algebra as e.g. GSLL. Is
anybody aware of that? Could I use this within shen code? Sorry for
the stupid question, I have no idea of Common Lisp so far.

Best regards
Johannes

[Mark Tarver]

Nested Shen vectors are not exactly CLarrays. however you can calls CL
arrays from Shen.

Mark

On May 13, 10:06 pm, Johannes Engels <johannes.eng...@hft-