Re: Shen 0.1 released

[Mark Tarver]

I think you mean the Shena database 0.1 program is released under BSD.
Shen is on version 11 now. :)

Mark

On May 28, 10:59 am, newbie <swtchwrd...@gmail.com> wrote:
> \* Shena is an experimental array langauge to learn about array programming
> in Shen. *\
> \* Shena 0.1 is released under BSD License. *\
>
> \*
>
> Copyright (C) 2013, W. Yang
>
> *** License:
>
> Redistribution and use in source and binary forms, with or without
> modification, are permitted provided that the following conditions are
> met:
>
>  - Redistributions of source code must retain the above copyright
>    notice, this list of conditions and the following disclaimer.
>
>  - Redistributions in binary form must reproduce the above copyright
>    notice, this list of conditions and the following disclaimer in the
>    documentation and/or other materials provided with the distribution.
>
> THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
> "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
> LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
> A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
> HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
> SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
> LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
> DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
> THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
> (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
> OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
>
> *\
>
> \* basic array functions *\
>
> \* (it0 20) gives [0 1 2 3 4 5 6 7 8 9 10... 18 19] *\
>
> (define it0
>
>      N -> (if (< N 0) [it0 takes a nonnegative number] (it N [])))
>
> (define it
>     0 L -> L
>     N L -> (it (- N 1) (cons (- N 1) L)))
>
> (define addto
>     N L -> (map (/. X (+ X N)) L))
>
> (define timesto
>     N L -> (map (/. X (* X N)) L))
>
> (define subtractfrom
>     N L -> (map (/. X (- N X)) L))
>
> (define divideby
>     _ L -> [divide by zero] where (= true (member? 0 L))
>     N L -> (map (/. X (/ N X)) L))
>
> \* useful when working with ShenRuby list in ruby interactive environment *\
> \* shen.it0(5) = [0,1,2,3,4] *\
> \* shen.hd1(shen.it0(5)) = 0 *\
>
> (define hd1
>      L -> (from 0 L))
>
> (define tl1
>      L -> (drop 1 L))
>
> \* member function, (member? 2 [1 2 3 4]) = true *\
>
> (define member?
>       _ [] -> false
>    N [X|Y] -> (if (= N X) true (member? N Y)))
>
> \* elementwise operation on two lists of same length *\
> \* (timesL [2 3][4 5]), (addL [2 3][4 5]), (subtractL [2 3][4 5]), (divideL
> [2 3][4 5]) *\
>
> (define timesL
>    _ [] -> []
>    _ [err] -> [[null multiply]]
>    _ [err1] -> [unequal length]
>    L M -> (timesL _ [err1]) where (not (= (length L) (length M)))
>    L M -> (if (or (= (hd L) null) (= (hd M) null)) (timesL _ [err])
> (reverse (timesL0 L M []))))
>
> (define timesL0
>       _ [] A -> A
>    [X|Y][P|Q] A -> (timesL0 Y Q (cons (* X P) A)))
>
> (define addL
>    _ [] -> []
>    _ [err] -> [[null add]]
>    _ [err1] -> [unequal length]
>    L M -> (addL _ [err1]) where (not (= (length L) (length M)))
>    L M -> (if (or (= (hd L) null) (= (hd M) null)) (addL _ [err]) (reverse
> (addL0 L M []))))
>
> (define addL0
>       _ [] A -> A
>    [X|Y][P|Q] A -> (addL0 Y Q (cons (+ X P) A)))
>
> (define subtractL
>    _ [] -> []
>    _ [err] -> [[null subtract]]
>    _ [err1] -> [unequal length]
>    L M -> (subtractL _ [err1]) where (not (= (length L) (length M)))
>    L M -> (if (or (= (hd L) null) (= (hd M) null)) (subtractL _ [err])
>                 (reverse (subtractL0 L M []))))
>
> (define subtractL0
>       _ [] A -> A
>    [X|Y][P|Q] A -> (subtractL0 Y Q (cons (- X P) A)))
>
> (define divideL
>    _ [] -> []
>    _ [err] -> [[null divide or divide by zero]]
>    _ [err1] -> [unequal length]
>    L M -> (divideL _ [err1]) where (not (= (length L) (length M)))
>    _ [P|Q] -> (divideL _ [err]) where (= P 0)
>    L M -> (if (or (= (hd L) null) (= (hd M) null)) (divideL _ [err])
> (reverse (divideL0 L M []))))
>
> (define divideL0
>       _ [] A -> A
>    [X|Y][P|Q] A -> (divideL0 Y Q (cons (/ X P) A)))
>
> \* few monadic functions *\
> \* (adds [2 3 4]) = 9, (times [2 3 4]) = 24, (divides [1 2 3]) = 1.5,
> (subtracts [1 2 3]) = 2 *\
>
> (define add0
>     [] N -> N
>    [X|Y] N -> (if (= X null) (add0 Y N) (add0 Y (+ N X))))
>
> (define adds
>      L -> (add0 L 0))
>
> (define times0
>     [] N -> N
>    [X|Y] N -> (if (= X null) (times0 Y N) (times0 Y (* N X))))
>
> (define times
>      L -> (times0 L 1))
>
> \* divides from right to left, such that (divides [1 2 3]) = (/ 1 (/ 2 3))
> = 1.5 *\
>
> (define divides0
>        _ N -> [divide by zero] where (= N 0)
>       [] N -> N
>    [X|Y] N -> (if (= X null) (divides0 Y N) (divides0 Y (/ X N))))
>
> (define divides
>      L -> (divides0 L 1))
>
> \* subtracts from right to left, such that (subtracts [1 2 3]) = (- 1 (- 2
> 3)) = 2 *\
>
> (define subtracts0
>     [] N -> N
>    [X|Y] N -> (if (= X null) (subtracts0 Y N) (subtracts0 Y (- X N))))
>
> (define subtracts
>      L -> (subtracts0 L 0))
>
> \* (take 2 [a b c d e]) = [a b], (take -2 [a b c d e]) = [d e], (take 0 [a
> b c d e]) = [] *\
> \* (take 20 (makearray 50 (it0 10000))) *\
>
> (define take0
>    0 M _ -> (reverse M)
>    N M [X|Y] -> (take0 (- N 1) (cons X M) Y))
>
> (define take
>     N L -> (if (< N 0) (drop (- (length L) (abs N)) L) (take0 N [] L)))
>
> \* (drop 2 [a b c d e]) = [c d e], (drop -2 [a b c d e]) = [a b c], (drop 0
> [a b c d e]) = [a b c d e] *\
> \* (drop 50 (makearray 100 (it0 10000))) *\
>
> (define drop
>       0 L -> L
>       N L -> (if (< N 0) (take (- (length L) (abs N)) L) (drop (- N 1) (tl
> L))))
>
> \* (fromto 2 5 [a b c d e f g]) = [c d e f] *\
>
> (define fromto
>     N M L -> (take (- (+ M 1) N) (drop N L)))
>
> \* (abs 5) = 5, (abs -5) = 5, (abs 0) = 0 *\
>
> (define abs
>    N -> (* N -1) where (< N 0)
>    N -> N)
>
> \* (from 2 [a b c d]) = c, (from 3 (makearray 5 (it0 200))) = [15 16 17 18
> 19] *\
>
> (define from
>    0 L -> (head L)
>    N [X|Y] -> (from (- N 1) Y))
>
> \* (insert 2 a [1 2 3 4]) = [1 2 a 3 4], (insert 2 (it0 5) [1 2 3 4]) = [1
> 2 [0 1 2 3 4] 3 4] *\
>
> (define insert
>    0 M L -> (append [M] L)
>    N M [X|Y] -> (append [X] (insert (- N 1) M Y)))
>
> \* (amend 2 a [1 2 3 4]) = [1 2 a 4], (amend 2 (it0 5) [1 2 3 4]) = [1 2 [0
> 1 2 3 4] 4] *\
>
> (define amend
>    0 M L -> (append [M] (tl L))
>    N M [X|Y] -> (append [X] (amend (- N 1) M Y)))
>
> \* (fromp [2 3 1] [a b c d e]) = [c d b] *\
>
> (define fromp0
>      [] _ A -> A
>   [X|Y] L A ->  (fromp0 Y L (cons (from X L) A)))
>
> (define fromp
>       L1 L2 -> (reverse (fromp0 L1 L2 [])))
>
> \* (intpart 23.3) = 23, (top 23.3) = 24, (base 23.3) = 23 *\
>
> (define intcount
>
>    N X -> N where (< X 1)
>    N X -> (intcount (+ N 1) (- X 1)))
>
> (define top
>    N -> (if (= N (intpart N)) N (+ (intpart N) 1)))
>
> (define base
>    N -> (intpart N))
>
> (define intpart
>    N -> (intcount 0 N))
>
> (define remainder
>    M N -> (if (< M N) (- N M) (- (/ M N) (intpart (/ M N)))))
>
> \* making an array *\
>
> \* (tie [2 3 4] 3) = [2 3 4 2 3 4 2 3 4] *\
> \* (tie (makearray 2 (makearray 5 (it0 100))) 3) *\
>
> (define tie
>         _ 0 -> []
>         L N -> (append L (tie L (- N 1))))
>
> \* "makearrayuneven" is for an irregular matrix *\
> \* fnull fills the last array with "null" to match the length of the rest
> of the arrays *\
>
> (define fnull0
>          0 L -> L
>          N L -> (fnull0 (- N 1) (cons null L)))
>
> (define fnull
>            N -> (fnull0 N []))
>
> \* (makearrayuneven1 3 (it0 10)) = [[0 1 2] [3 4 5] [6 7 8] [9 null null]]
> *\
>
> (define makearrayuneven1
>        0 N L -> (append (reverse N) [(append L (fnull (- (length (hd N))
> (length L))))])
>        M N L -> (if (< M (length L)) (makearrayuneven1 M (cons (take M L)
> N) (drop M L))
>                   (makearrayuneven1 0 N L)))
>
> (define makearrayuneven
>         M _ -> [] where (= M 0)
>         M L -> (makearrayuneven1 M [] L ))
>
> (define makearray1
>        0 N L -> (append (reverse N) [L])
>        M N L -> (if (< M (length L)) (makearray1 M (cons (take M L) N)
> (drop M L))
>                   (makearray1 0 N L)))
>
> (define makearray
>         M _ -> [] where (= M 0)
>         M L -> [try "makearrayuneven", but some functions may not work
> properly.]
>                                 where (not (= 0 (remainder (length L) M)))
>         M L -> (makearray1 M [] L))
>
> \* The function "makearray" with "tie" function can create a single,
> continous, connected database capable
>
>    of holding hundreds of billions of data, but doing so can destabilize
> the system resulting in stack overflow
>    or access violation.
>
>    The following creates an array (aa3) with close to 100 billion
> placeholders for data input.
>
>     (define aa -> (makearray 20 (makearray 250 (it0 40000))))
>         (define aa1 -> (tie (aa) 650))
>      (define aa2 -> (tie (aa1) 180))
>     (define aa3 -> (tie (aa2) 20))
>
>    Here's another approach:
>
>         (define a -> (makearray 5 (it0 10000))) <-- a = [[0 1 2 3 4] [5 6 7
> 8 9]...[9995 9996 9997 9998 9999]]
>         (define b -> (fromto 2000 3999 (makearray 5 (it0 20000)))) <-- b =
> [[10000 10001..].. [.. 19999]]
>         (define c -> (makearray 5 (addto 20000 (it0 10000)))) <-- c =
> [[20000 20001 20002..] .. [.. 29999]]
>
>         (define myarray -> [a b c])
>
>    Each element in (myarray) = [a b c] is a single character, a, b, c, but
> calling any one of them ("a",
>    for instance) with matching parentheses activates the evaluation of the
> array denoted by "a",
>
>         ((from 0 (myarray))) = (a) = the array represented by "a" as
> defined above.
>
>    This allows calculation of only those arrays in (myarray) that the query
> function is actually
>    working on, thus freeing the system resources for other tasks at hand.
> *\
>
> \* (ezq 25000 (myarray)) = 25000 -- picks the 5000th item from database (c)
>
>    (ezqx 5500) = 5500 (5500th data in the 1st database a), (ezqx 17000) =
> 7000, (7000th data in the 2nd
>
>                  database b) *\
>
> (define ezqx
>        N -> (- N (* (intpart (/ N 10000)) 10000)))
>
> (define ezq
>       N L -> (from (ezqx N)
>                      (cleanall ((from (intpart (/ N 10000)) L)) )))
>
> \* The function clean removes the square brackets one pair at a time,
> cleanall removes them all.
>    clean and cleanall only work with numbers.
>        (clean [1 [2 [100 200] 3]]) = [1 2 [100 200] 3], (cleanall [1 [2
> [100 200] 3]]) = [1 2 100 200 3] *\
>
> (define clean
>       [] -> []
>    [X|Y] -> (if (number? X) (append [X] (clean Y)) (append X (clean Y))))
>
> (define cleanall
>       L -> (if (= (clean L) L) L (cleanall (clean L))))
>
> \* rows and columns *\
>
> \* The row function as defined starts counting from row 0, row 1,... *\
>
> (define row
>        N L -> (from N L))
>
> \* (define a -> (makearray 5 (it0 25)))
>    (amendrow 3 [a b c d e] (a)), (amendrow 3 [a b c] (a)), (amendrow 2
> (makearray 5 (it0 25)) (a))
>    (insertrow 3 [a b c d e] (a)), (insertrow 3 [a b c] (a))
>    (deleterow 2 (a)), (showrow 1 (a)) *\
>
> (define amendrow
>      N M L -> (if (= (length M) (length (hd L))) (amend (- N 1) M L)
> [unequal length]))
>
> (define addrowtop
>        M L -> (if (= (length M) (length (hd L))) (append [M] L) [unequal
> length]))
>
> (define addrowdown
>        M L -> (if (= (length M) (length (hd L))) (append L [M]) [unequal
> length]))
>
> (define insertrow
>      N M L -> (if (= (length M) (length (hd L))) (insert (- N 1) M L)
> [unequal length]))
>
> (define deleterow
>        N L -> (append (take (- N 1) L) (drop N L)))
>
> (define showrow
>        N L -> (from (- N 1) L))
>
> \* nth column from matrix L, (col 2 (makearray 5 (it0 20))) = [2 7 12 17] *\
> \* (adds (col 2 (makearray 5 (it0 20)))) = 38 *\
> \* (define a -> (makearray 5 (it0 50000))) *\
> \* (adds (addL (col 1 (a)) (col 3 (a)))) = 499990000 *\
>
> (define col
>       N L -> (reverse (col0 N L [] (it0 (length L)))))
>
> (define col0
>      _ _ M [] -> M
>   N L M [X|Y] -> (col0 N L (cons (from N (row X L)) M) Y))
>
> \* (define a -> (makearray 5 (it0 20))) = [[0 1 2 3 4] [5 6 7 8 9] [10 11
> 12 13 14] [15 16 17 18 19]] *\
> \* (colgo (a) (it0 2)) = [[0 5 10 15][1 6 11 16]] *\
>
> (define colgo1
>      _ M [] -> M
>      L M [X|Y] -> (colgo1 L (cons (col X L) M) Y))
>
> (define colgo
>         L L1 -> (reverse (colgo1 L [] L1)))
>
> \* samelength returns true if the elements in a list are of the same size,
> otherwise returns false. *\
>
> (define samelength
>         [] -> true
>          L -> (samelength []) where (< (length L) 2)
>      [X|Y] -> (if (= (length X) (length (hd Y))) (samelength Y) false))
>
> \* mflip takes arrays of numbers, characters, words, arrays, arrays of
> arrays *\
> \* (define a -> (makearray 5 (it0 25))), (mflip (a)) *\
>
> (define mflip
>     [err] -> [unequal length]
>         L -> (if (not (samelength L)) (mflip [err]) (colgo L (it0 (length
> (hd L))))))
>
> \* addcol adds a column to the rightmost side of an array *\
> \* (define a -> (makearray 5 (it0 25))), (addcol (a) [a b c d e]) *\
> \* (addcolleft (a) [a b c d e]), (addcolright (a) [a b c d e]) *\
> \* (amendcol 2 [a b c d e] (a)), (insertcol 3 [a b c d e] (a)), (showcol 2
> (a)) *\
>
> (define addcol
>       L M -> (mflip (append (mflip L) [M])))
>
> (define addcolleft
>       L M -> (if (= (length M) (length (hd L))) (mflip (addrowtop M (mflip
> L))) [unequal length]))
>
> (define addcolright
>       L M -> (if (= (length M) (length (hd L))) (addcol L M) [unequal
> length]))
>
> (define amendcol
>     N M L -> (if (= (length M) (length (hd L))) (mflip (amend N M (mflip
> L))) [unequal length]))
>
> (define insertcol
>     N M L -> (if (= (length M) (length (hd L))) (mflip (insert N M (mflip
> L))) [unequal length]))
>
> (define showcol
>       N L -> (col (- N 1) L))

[Raoul Duke]

>> (define it0
>> N -> (if (< N 0) [it0 takes a nonnegative number] (it N [])))
>

> Isn't the common name for something like it0 "iota"? Also, I noticed a trend

er.

no, i don't think so. :-)

e.g. random link http://www.jb.man.ac.uk/~slowe/cpp/itoa.html

[Pierpaolo Bernardi]

The greek letter iota for this function has been introduced by APL,
and many languages adopted the name 'iota' for this funtion.

The C function itoa is a completely different matter. :)

[Martial B]

Not really, the original APL 'iota' is lazy (also used in factor; and unorthodoxly in Go). It doesn't expand as it0.

Cheers,

--
Martial

[Mark Tarver]

This might help.

http://www.shenlanguage.org/learn-shen/system.pdf#system-functions

Mark

On May 29, 3:57 am, newbie <swtchwrd...@gmail.com> wrote:
> \* Querying (row 2 [a list]) gives the row 1 instead of row 2.
>    I could either use from with 1-indexing
>
>     (define from1
>          1 L -> (hd L)
>          N L -> (from1 (- N 1) (tl L)))  *\
>
> I meant that (row 2 [a list of array]) gives the row 3 instead of row 2
> as supposed to be per row numbering convention.
> I haven't had much sleep over the weekend, so I probably need to
> sleep a little more so I can keep my thoughts straight.

[Mark Tarver]

The question of vectors vs lists for data processing is indeed an
interesting one. There are advantages on both sides.

First as regards writing a vector-handling program vs a list-handling
program, they are equally easy in Shen. List-matching, string-
matching and vector-matching are all in the language.

http://www.shenlanguage.org/learn-shen/functions/functions_pattern_matching.html

The advantage of vectors is that the operation of accessing the nth
element is constant time whereas for the list it is linear time.
Supposing that the vector is sorted or your query is driven by
hashing, this gives you a very good performance. However when it
comes to consing, the position is exactly reversed. Consing is a
linear time operation for vectors in Shen requiring copying out the
vector, but constant time in lists. Hence vector manipulation
programs in Shen using @v are slow and so are string operations using
@s because in CL strings are vectors of characters. Therefore though
@v and @s allow you to write very elegant, clear programs, large
strings or vectors are expensive to process that way in Shen. In
fact, for string processing in the large, like reading a large file,
it is better to read the file to a list of unit strings. Experiments
have shown that processing a 250K file has a speedup of 10X using this
technique.

If you look in the vector library in standard lib, you'll find all the
vector functions are '@v' free for speed and use the lower level
operations that Shen provides.

Regarding your work, certainly lists have an advantage in processing
but not in look up. There are several possibilities here. One is to
switch to binary trees using lists, which reduces look up to log
time. Another is to use vectors but process answers in terms of
lists.

Last, at some point you will have to come to grips with types.

Mark

On 30 May, 06:10, newbie <swtchwrd...@gmail.com> wrote:
> Show me that you can use vectors to do this and my curiosity will be piqued:
>
> (timesL (makearray 5 (makearray 50 (makearray 10 (it0 20000)))) (makearray
> 5 (makearray 50 (makearray 10 (addto 2 (it0 20000))))))
>
> (subset (makearray 5 (it0 40000)) (makearray 5 (it0 4000000)))
>
> (intersect (makearray 20 (it0 1000000)) (mkaearray 20 (timesto 5 (it0
> 1000000))))
>
> Instead of worrying about that, you could use the Shena database functions
> to create "ascend" and "descend" functions (in addition to auxiliary
> "greater than", "less than" functions, such that
>
>     (ascend [10 1 3 50 2]) = [1 2 3 10 50]
>     (descend[10 1 3 50 2]) = [50 10 3 2 1]
>
> then the "range" (range over the whole of the list, not the iterate) would
> give
>
>     (define range
>            L -> (- (hd (reverse (ascend L))) (hd (ascend L))))
>
> What could be simpler?

[newbie]

Hi,

Thank you for suggesting the package function.
It should be good to use as the program gets bigger.

I was just remembering how you said that Dr. Tarver
suggests using hyphens between long names, it does
make a difference in readibility, even the underscore.
I don't like to type long confusing-looking names especially
when I'm trying to make a function work. I actually abbreviated
make-array to "ma" myself when I was testing them out.
I guess I was a little hyper in my response due to having a long
day on very little sleep and not enough coffee (or maybe too much of
it)...

I know that make-array works fine in ShenRuby but it failed to work
when I was creating arrays at the interactive prompt by typing
something
like... (after loading the file with (load "C://myfile.shen))

shen.make-array(5, shen.it0 (500))

but shen.ma(5,shen.it0 (500)) worked.

but that was some time ago and maybe the new version doesn't have that
situation.

I want to follow "write once, run everywhere" concept as much as
possible.
I deleted all the symbols from function names too.
but while (it0 20000000) works in Shen, the upper limit lowers in
other Shen phenotypes.

"repeat" is a nice function.
Thank you for that.

Please check out my (bigdata) post under the vector challenge heading.
It uses "from" which is O(1) but it uses it to bypass the unnecessary
arrays within (bigdata) list.
It picks up only the arrays that contain the answer for the search
query (simple position query,
what is the data value in position X). If the query asked for minimum/
maximum in a database,
it would go back to O(1).

It can find the answer in a minute and a half or less no matter the
size of the entire database (currently possible at 100~200 trillion
per database.)

and one can always create a metadatabase, or meta-database comprised
of elements like

(define meta-database -> [bigdata1 bigdata2 bigdata3...
bigdata19999999 bigdata20000000])

where each (bigdata1), (bigdata2)... contains 100~200 trillion data
(simple numbers, short words)

and for simple search for value retrieval or amend, insert, delete,
the search time would still be under a minute
and a half for the particular configuration of (bigdata) having a size
of only 5 million per element (if it's increased to
10 million, it may take more than 2 minutes, I suppose).

But, I know that O(N) is a very valuable notion.
I'm reading up on it.

Thank you.