Installation

[David Hodge]

Dear all,

Here is my latest README.org for the installation chapter for review.

As a note, my aim is to make the installation process as smooth as possible, bearing in mind the changes that CLS is likely to undergo.

So my approach is to separate out libraries already in quick lisp and just leave the CLS specific stuff within the repo.

This means we get down from 12 external dependancies to about 3 or 4.  By external I mean other libraries present in the "external" directory - lisp-matrix being the major tenant there.

So the dependancies in external have gone from 

fnv

data-table

lla

xarray

listoflist

lisp-matrix

rsm-string-cls

cl-blapack

lift

cldoc

ffa-lisp-matrix

array-operations-lisp-matrix

to 

cl-blapack (in quicklisp, need to push)

data-table (in quick lisp, not moved as yet. dunno where used)

listoflist 

lisp-matrix

rsm-string-cls (optional utility, cesarum or similar in ql already)

ffa-lispmatrix (also part of lisp-matrix I think)

array-operations-lisp-matrix (??)

cl-blapack is still there, bacuase of a slight mod I have made which means it just installs without requiring modification. I should push that to tony and then it wold end up being in quick lisp

array-operations-lispmatrix is purely part of lisp-matrix and does not in my opinion deserve to be a separate library

rsm-string is  utility  library and frankly should not be here, there are a bunch of equivalent things already in quick lisp i think  so, in my mind, it should disappear in favour of one of them eventually.

my day job is in the java world where dependancy management is often a nightmare and i would like to make things as simple as possible in the future.

I actually think that using something like asdf-system-connections for optional or value add packages would be something to consider.

Having said that, here is the latest version of the install instruction

==========================

 Fast Start

Common Lisp Stat is a very rich and mature framework for data

exploration and visualisation using statistics with the Common Lisp

Environment.

This design choice gives a number of advantages - both in the ability

to create fast running code, as well as being able to express complex

algorithms in a powerful and consistent language.

However, as with all large complex things, setup requires a few steps.

We assume that you have a lisp installed and that you have a passing

acquaintence with the unix shell.

1. The first point that you should note that is that these instructions

are written with the assumption of the availibility of quicklisp.

If you do not have quicklisp , please go to www.quicklisp.org and

install it now

2. The second point to note is that you will need the "git" utility

installed on your machine.

for mac osx sudo port install git

for linux (eg debian) sudo apt-get install git

3. Once that is done execute the following shell commands

#+begin_src shell

  cd ~/quicklisp/local-projects

  git clone git://github.com/blindglobe/common-lisp-stat.git

  cd common-list-stat

  git submodules init

#+end_src

These commands copy the the source from the repository and all the

associated libraries. It will live as a quicklisp project in the

local-projects directory. I find it convenient to symbolically link the

quicklisp direct to ~/lisp for easy access

#+begin_src shell

   ln ~/quicklisp/local-projects ~/lisp

#+end_src

4. Configure the locations of the BLAS and LINPACK libraries

For MacOS, we will use the pre installed libraries

For linux, make sure you have the neccessary libraries installed,

through apt, yum or otherwise viz

viz sudo apt-get install libblas, sudo apt-get install liblapack

5. For visualization we are currently implementing a gnuplot

   interface, stay tuned.

For windows, we recommend you use cygwin to get straightforward

access. I'll document the steps if there is a demand.

6. You need to check that your dynamic library path has been properly

   set up in the shell.

In your .bashrc (or equivalent shell init file) 

For Mac OSX set DYLD_FALLBACK-LIBRARY_PATH=$DYLD_FALLBACK_LIBRARY_PATH:/opt/local/lib

For Linux set LD_LIBRARY_PATH=$LD_LIBRARY_PATH:????

If you get this wrong the load process will not be able to find the

libraries and will prompt you. 

5. Once the pre prequisites have been done, start your favourite lisp

   and enter 

#+begin_src lisp

(ql:register-local-projects)

(ql:quickload :cls) 

#+end_src lisp

Retire for a well earned coffee and upon your return you should find

the package completely installed.Obviously, potential errors can creep

in with spelling the filenames correctly, so be careful.

[A.J. Rossini]

Hi David -

I think that the only difference that I have between what I've been doing and with your approach is that I'm trying to get rid of submodules and move them into standalone quicklisp-fetchable packages.

But the write-up you provided is an invaluable starting point!

best,
-tony

[David Hodge]

Hi,

Well, in fact tHere is no difference in intent, just implementation :)

While i would very much like to get rid of the external directory, my thinking was that some things like lisp matrix are likely to change pretty dramatically over the next x ( x> 3) months and it just  makes practical sense to have it.

Ultimately, external directory should disappear , I absolutely agree.

However, as we approach release 0.09 ( sometime in the middle distant future) being able to push changes to critical libraries quickly will be important IMO.

I could be wrong, and am willing to to listen to alternative point of view of course. But it's not as important  to me as data frames and data imports for instance

Cheers

Sent from my iPad

On 4 Nov, 2012, at 3:29 PM, "A.J. Rossini" <blind...@gmail.com> wrote:

Hi David -

I think that the only difference that I have between what I've been doing and with your approach is that I'm trying to get rid of submodules and move them into standalone quicklisp-fetchable packages.

But the write-up you provided is an invaluable starting point!

best,
-tony

On Sunday, November 4, 2012 1:48:09 AM UTC+1, David Hodge wrote:

Dear all,

Here is my latest README.org for the installation chapter for review.

As a note, my aim is to make the installation process as smooth as possible, bearing in mind while the changes that CLS is likely to undergo.

--
You received this message because you are subscribed to the Google Groups "Common Lisp Statistics" group.
To post to this group, send email to lisp...@googlegroups.com.
To unsubscribe from this group, send email to lisp-stat+...@googlegroups.com.
Visit this group at http://groups.google.com/group/lisp-stat?hl=en.
 
 

[A.J. Rossini]

Actually, lisp-matrix should be relatively stable on a monthly basis.
Stuff to do there, but I think I need to get it and the dependencies
properly vetted from quicklisp. The big need right now is a nice
front end to some of the LAPACK routines. But in theory, that can be
written where it is needed in CLS, and migrated over to lisp-matrix as
it gets upgraded.

Agree with the "push quick changes out" thinking, though -- am just
not sure what qualifies at this point. I think I did factor out
RSM-STRING in my version, and will just need to include comments on
how to get it if one wants to use it, in the particular
examples/*.lisp files -- but as I mentioned earlier, am working to get
this dependency removed in favor of a quicklisp'able csv-slurper.

--
best,
-tony

blind...@gmail.com
Muttenz, Switzerland.
"Commit early,commit often, and commit in a repository from which we
can easily roll-back your mistakes" (AJR, 4Jan05).

Drink Coffee: Do stupid things faster with more energy!

[Peter Schmiedeskamp]

This may already be common knowledge, but attempts to load lisp-matrix fail for me on SBCL for MacOS X (installed via homebrew). 

Specifically, things barf on the the unittests.lisp:

CL-USER> (ql:quickload "lisp-matrix")

To load "lisp-matrix":

  Load 1 ASDF system:

    lisp-matrix

; Loading "lisp-matrix"

..................................................

[package lisp-matrix].............................

[package lisp-matrix-user]........................

[package lisp-matrix-unittests]...................

...............................................

; 

; caught ERROR:

;   READ error during COMPILE-FILE:

;   

;     :ASCII stream decoding error on

;     #<SB-SYS:FD-STREAM

;       for "file /Users/peter/quicklisp/local-projects/lisp-matrix/src/unittests/unittests.lisp"

;       {1008C74FF3}>:

;   

;       the octet sequence #(226) cannot be decoded.

;   

;     (in form starting at line: 224, column: 34, file-position: 7691)

I found a page that suggests the problem is that SBCL on the Mac defaults to reading things in ASCII mode, and we have some unicode characters in that file (at least in Tony's branch--I haven't verified anything upstream). https://github.com/mon-key/unicly/issues/1

One quick work-around appears to be to put the following in my ~/.sbclrc

#+sbcl (unless (eq sb-impl::*default-external-format* :UTF-8)

(setf sb-impl::*default-external-format* :UTF-8))

Also, after some effort, I got CLS set up and loading via slightly different means. I have no idea if my method produces a working version, but everything at least loads cleanly via quicklisp. I'll do some playing with examples and then clean up the documentation of what I did. In short, I avoided using the "external" directory in favor of creating my own forks in github, and then cloning forks of relevant libraries to my local-projects directory. The (optimistic) idea being that if I make changes to any of the supporting packages, I can simply push my changes back to github and then issue a pull request. Please let me know if this is a bad approach!

Cheers,

Peter

[David Hodge]

Hi,

I am developing on a mac right now, and have not run across this.

I have a recent copy of Tony's repo and the end of the file looks fine to me. the # character and two linefeeds.

Can you verify the same for the version of the file you have? 

From curiosity what is the version of sbcl you are using?

[A.J. Rossini]

On Sunday, November 4, 2012 5:52:09 PM UTC+1, Peter Schmiedeskamp wrote:

This may already be common knowledge, but attempts to load lisp-matrix fail for me on SBCL for MacOS X (installed via homebrew). 

Which version SBCL?
 

Externals should be nuked.  Hopefully doesn't exist, please pull the latest changes (latest was yesterday).

The general approach just having a forked local repositoryshould be fine, using the following approach:

# first, fork on github, assume your user name is "IuseThis"

git clone  git://github.com/IuseThis/common-lisp-stat.git
cd common-lisp-stat
git remote add canonical-version git://github.com/blindglobe/common-lisp-stat.git
git checkout -b local
<work, fix, work, etc>
git fetch --all

(and compare current status with canonical-version, occassionally doing a
    git rebase canonical-version/master
to get things synced with the so-called "canonical-version")

This way you have a local fork I can pull from, and in addition, you can directly pull from my copy and yours as you want to.

best,
-tony

[David Hodge]

I absolutely agree, external directory should disappear over time. the sooner the better

I will implement the git canonical thingy asap.

[Peter Schmiedeskamp]

Tony and David,


>
> Which version SBCL?
>

1.0.58, but I see 1.1.1 is in homebrew as of sometime in the last day
or two. I'll try upgrading that today, without the modification to my
.sbcl, to see if whatever the issue is has been fixed upstream. I
should say I'm assuming there were some unicode characters in the
file, as adding the line to force sbcl to use unicode by default
seemed to fix the problem for me. It's possible that SBCL installed
via other means (than homebrew) sets this option by default?

>
>
> Externals should be nuked. Hopefully doesn't exist, please pull the latest
> changes (latest was yesterday).
>
> The general approach just having a forked local repositoryshould be fine,
> using the following approach:
>
> # first, fork on github, assume your user name is "IuseThis"
>
> git clone git://github.com/IuseThis/common-lisp-stat.git
> cd common-lisp-stat
> git remote add canonical-version
> git://github.com/blindglobe/common-lisp-stat.git
> git checkout -b local
> <work, fix, work, etc>
> git fetch --all
>
> (and compare current status with canonical-version, occassionally doing a
> git rebase canonical-version/master
> to get things synced with the so-called "canonical-version")
>
> This way you have a local fork I can pull from, and in addition, you can
> directly pull from my copy and yours as you want to.
>
> best,
> -tony

That's almost exactly what I ended up doing (I called you "upstream"
as opposed to "canonical-version"). My "set everything up" shell
script:

#!/bin/bash
# First create a private fork of blindglobe's relevant CLS repos via
# the web:
# 1. common-lisp-stat
# 2. lisp-matrix
# 3. listoflist
# 4. rsm-string-cls

# Set this appropriately
cd ~/quicklisp/local-projects

# Your github username
username=pschmied

git clone g...@github.com:$username/common-lisp-stat.git
cd common-lisp-stat
git remote add upstream git://github.com/blindglobe/common-lisp-stat.git
git checkout -b local
cd ..

git clone g...@github.com:$username/lisp-matrix.git
cd lisp-matrix
git remote add upstream git://github.com/blindglobe/lisp-matrix.git
git checkout -b local
cd ..

git clone g...@github.com:$username/listoflist.git
cd listoflist
git remote add upstream git://github.com/blindglobe/listoflist.git
git checkout -b local
cd ..

git clone g...@github.com:$username/rsm-string-cls.git
cd rsm-string-cls
git remote add upstream git://github.com/blindglobe/rsm-string-cls.git
git checkout -b local
cd ..

# git clone g...@github.com:$username/cl-blapack.git
# cd cl-blapack
# git remote add upstream g...@github.com:blindglobe/cl-blapack.git
# cd ..


echo -e "Don't forget to edit your
~/quicklisp/dists/quicklisp/software/cl-blapack-20120811-git/load-blapack-libs.lisp"
echo -e "Now, start your common lisp and
invoke:\n(ql:register-local-projects) and (ql:quickload :cls)"

[Peter Schmiedeskamp]

Well, it looks like with SBCL 1.1.1, we now run afoul of "package locks" (http://www.sbcl.org/manual/Package-Locks.html) when compiling org.middleangle.foreign-numeric-vector:

Lock on package SB-INT violated when proclaiming

SB-INT:BLOCK-GENSYM as a function while in package

COMMON-LISP-USER.

   [Condition of type SYMBOL-PACKAGE-LOCKED-ERROR]

See also:

  SBCL Manual, Package Locks [:node]

If I elect to ignore the package lock and continue, everything appears to load per normal. As to the first error, I don't see that now, but perhaps I need to recompile everything?

Cheers,

Peter

[A.J. Rossini]

On Mon, Nov 5, 2012 at 4:02 PM, Peter Schmiedeskamp
<pe...@thoughtspot.net> wrote:
> Well, it looks like with SBCL 1.1.1, we now run afoul of "package locks"
> (http://www.sbcl.org/manual/Package-Locks.html) when compiling
> org.middleangle.foreign-numeric-vector:
>
> Lock on package SB-INT violated when proclaiming
> SB-INT:BLOCK-GENSYM as a function while in package
> COMMON-LISP-USER.
> [Condition of type SYMBOL-PACKAGE-LOCKED-ERROR]
> See also:
> SBCL Manual, Package Locks [:node]
>
>
> If I elect to ignore the package lock and continue, everything appears to
> load per normal. As to the first error, I don't see that now, but perhaps I
> need to recompile everything?

Thanks, I've also got a bug report from Xach (quicklisp guy) on
cl-blapack reporting on the block-gensym issue, and I'm guessing that
it is the same. It looks like SBCL 1.x introduced a few changes in
the name of progress that will require some debugging for cl-blaplack.
Just when I didn't want an interruption...