Element-wise Math in CL, part 2

[Steven Núñez]

Happy New Year,

 

I spent the last few days of my holiday looking extracting the element-wise math functions out of CLS, with mixed success. I did manage to remove the compound data object code. It seems that xlisp stat never really implemented compound data as xlisp objects, so it was easy to remove them.

 

The four basic arithmetic operators are all working, as well as a few other functions, but all the rest are broken. For some reason, trying to use make-rv-function on something like sqrt produces a recursive error. I’ve been banging my head on this for 4 days now, but no success, though I think it must be close. My guess is that it is somewhere in the map-element* functions.

 

Question for those of you with a working common lisp stat: do functions other than the basic ones work for you? Can you do (sqrt ‘(1 2 3)) successfully?

 

This stuff takes a lot of time to understand. Can any of you common lisp stat gurus tell me what I’ve done wrong? I’ve put what I’ve got up on github at https://github.com/senor-hadoop/clsm, which isolates the code to just the element-wise math functions, as well as cleans up and organises some of the files.

 

All the best in the new year,

 

Regards,

-          SteveN

 

 

 

.

[David Hodge]

Hi Steve,

Let me have a look at my copy of CLS later tonight.

I know nothing of compound data types  - but last time I looked I thought that the element stuff worked after a fashion.

Cheers

Steven Núñez

5 January 2015 12:25 pm

--
You received this message because you are subscribed to the Google Groups "Common Lisp Statistics" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lisp-stat+...@googlegroups.com.
To post to this group, send email to lisp...@googlegroups.com.
Visit this group at http://groups.google.com/group/lisp-stat.
For more options, visit https://groups.google.com/d/optout.

[David Hodge]

I had 5 minutes to quickly look

Loading CLS from quicklisp worked (yay!)

in cls user I did

LS-USER> (sqrt '(1 2 3))
(1.0 1.4142135623730951 1.7320508075688772)

likewise 

(+ 1 '(4 5 6))
(5 6 7)

and

LS-USER> (^ 2 '(2 4 6))
(4 16 64)


so prima facie these things work. I would however expect its very sensitive to correct interning of symbols and other such mysteries.

What exactly is the error you are getting?

Steven Núñez

5 January 2015 12:25 pm

Happy New Year,

--

[Steven Núñez]

I’m getting an infinite loop. This might be because of my tinkering, though I was rather careful in the extraction.

What lisp are you using? I was trying it with CCL.

Quicklisp? I read that announcement, but thought that meant fiddling with lisp matrix and friends. Is it really as easy as (ql:quickload :cls) ?

Cheers,

- Steve

[A.J. Rossini]

Yes, quicklisp should work and then you can step through the files in the examples or the top level xlispstat old example which should work. 

Start of (school) year has kept me busy but I have a few commits for rho , CLS.  Good news in that someone is fixing cl-random to have a platform independent reproducible RNG, though that part is not quite done and the while package brings in a huge load of others....

[A.J. Rossini]

And I will take a look at your stuff Steve tomorrow. 

[Steven Núñez]

Hi Tony,

Thanks. Any luck? I’m going be using the math stuff at the end of Jan and, if it can run on CCL/MacOS, would prefer to do it on lisp stat than R.

David, what platform were your test run?

Regards,

- Steve

[David Hodge]

Hey Steve,

I did mine on on macosx and a recent version of sbcl (1.24 I think)

I tried to clone your repo but it's ccl specific  it seems. I will hopefully have some time to install and configure ccl tonight and look in a little more depth.

Is there a reason for the package-inferred asdf stuff? 

Cheers

Sent from my iPad

<compose-unknown-contact.jpg>

[Steven Núñez]

Hi Dave,

There’s only one place there’s a CCL specific compile directive; it’s because in CCL numberp is a built-in, so it needed to be removed.

No reason in particular for the packaged inferred stuff. It seemed like a good idea from a software engineering perspective, and since I was extracting the math stuff anyway thought I’d refactor a bit as well.

Regards,

- Steve

[David Hodge]

Hi Steve,

There is actually a package called :ccl mentioned in compound.lisp and math.lisp. There is also a ccl specific variable being set in math.lisp. which should be attdend to.

So, when I compile this i get package lock violations for all the operators

When I do something like
(1+ '(1 2 3))

I get a boat load of debugging and then the appropriate answer eventually.

the debugging looks like

In make-rv-function with 1+ In cmpndp with (1
followed by a very long blank line , then "in cmpndp with (2 , another blank line, then the same for 3.

The right answer is returned, but without understanding what  the debugging is for, its hard for me to say whats happening.
I'll read through the code after the kids are in bed and try to see whats happening.

At a guess i would suggest that you turn the debugging off, or direct it into file and see if it l completes

Cheers


I would suggest

Steven Núñez

7 January 2015 8:21 pm

David Hodge

7 January 2015 6:40 pm

Steven Núñez

7 January 2015 6:34 pm

Hi Tony,

Thanks. Any luck? I’m going be using the math stuff at the end of Jan and, if it can run on CCL/MacOS, would prefer to do it on lisp stat than R.

David, what platform were your test run?

Regards,

- Steve

From: "A.J. Rossini" <blind...@gmail.com>
Reply-To: "lisp...@googlegroups.com" <lisp...@googlegroups.com>
Date: Sunday, 4 January 2015 22:21
To: lisp-stat <lisp...@googlegroups.com>
Subject: Re: [lisp-stat] Element-wise Math in CL, part 2

And I will take a look at your stuff Steve tomorrow. 

--

You received this message because you are subscribed to the Google Groups "Common Lisp Statistics" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lisp-stat+...@googlegroups.com.
To post to this group, send email to lisp...@googlegroups.com.
Visit this group at http://groups.google.com/group/lisp-stat.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Common Lisp Statistics" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lisp-stat+...@googlegroups.com.
To post to this group, send email to lisp...@googlegroups.com.
Visit this group at http://groups.google.com/group/lisp-stat.
For more options, visit https://groups.google.com/d/optout.

A.J. Rossini

5 January 2015 7:21 pm

And I will take a look at your stuff Steve tomorrow. 

--

You received this message because you are subscribed to the Google Groups "Common Lisp Statistics" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lisp-stat+...@googlegroups.com.
To post to this group, send email to lisp...@googlegroups.com.
Visit this group at http://groups.google.com/group/lisp-stat.
For more options, visit https://groups.google.com/d/optout.

A.J. Rossini

5 January 2015 7:21 pm

Yes, quicklisp should work and then you can step through the files in the examples or the top level xlispstat old example which should work. 

Start of (school) year has kept me busy but I have a few commits for rho , CLS.  Good news in that someone is fixing cl-random to have a platform independent reproducible RNG, though that part is not quite done and the while package brings in a huge load of others....

--

[David Hodge]

I should also mention that I can compile mod & rem perfectly well (modulo the comments about package locks)


Running a simple (1+ '(1 2 3) seems to generate a lot of calls to cmpndp which appear to me to be overhead. Let me try to wrap my head around the macro and understand what its doing.

Of course, do you really need compound data? whats the use case?

Cheers

Steven Núñez

7 January 2015 8:21 pm

David Hodge

7 January 2015 6:40 pm

Steven Núñez

7 January 2015 6:34 pm

Hi Tony,

Thanks. Any luck? I’m going be using the math stuff at the end of Jan and, if it can run on CCL/MacOS, would prefer to do it on lisp stat than R.

David, what platform were your test run?

Regards,

- Steve

From: "A.J. Rossini" <blind...@gmail.com>
Reply-To: "lisp...@googlegroups.com" <lisp...@googlegroups.com>
Date: Sunday, 4 January 2015 22:21
To: lisp-stat <lisp...@googlegroups.com>
Subject: Re: [lisp-stat] Element-wise Math in CL, part 2

And I will take a look at your stuff Steve tomorrow. 

--

You received this message because you are subscribed to the Google Groups "Common Lisp Statistics" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lisp-stat+...@googlegroups.com.
To post to this group, send email to lisp...@googlegroups.com.
Visit this group at http://groups.google.com/group/lisp-stat.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Common Lisp Statistics" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lisp-stat+...@googlegroups.com.
To post to this group, send email to lisp...@googlegroups.com.
Visit this group at http://groups.google.com/group/lisp-stat.
For more options, visit https://groups.google.com/d/optout.

A.J. Rossini

5 January 2015 7:21 pm

And I will take a look at your stuff Steve tomorrow. 

--

You received this message because you are subscribed to the Google Groups "Common Lisp Statistics" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lisp-stat+...@googlegroups.com.
To post to this group, send email to lisp...@googlegroups.com.
Visit this group at http://groups.google.com/group/lisp-stat.
For more options, visit https://groups.google.com/d/optout.

A.J. Rossini

5 January 2015 7:21 pm

Yes, quicklisp should work and then you can step through the files in the examples or the top level xlispstat old example which should work. 

Start of (school) year has kept me busy but I have a few commits for rho , CLS.  Good news in that someone is fixing cl-random to have a platform independent reproducible RNG, though that part is not quite done and the while package brings in a huge load of others....

--

[Steven Núñez]

Hi Dave,

The :ccl in the package statements are for the numberp and for the warning flag. If you set the *ccl-kernel-warning* to nil then you won’t get the package locks.

That print statement is in compound.lisp, first function (I don’t have the code in front of me and am doing this from memory, so it might not be exactly the first, but it’s at the top). It’s not useful now, since the problem appears to be in map-elements and map-recursive elements. This might just be a simple matter of the recursion not ending.

On the git hub page I listed all the operators that do work. The basic math functions are all good; the functions that Tony wrapped in floats.lisp all fail. It is easy to turn on/off those wrapped functions; float.lisp just handles single->double floats by wrapping each function in a base_function-name function that’s later passed to make-rv-function.

I guarantee failure by trying (sqrt ‘(1 2 3)) or (sin ‘( 1 2 3)), but they have to be uncommented in math.lisp. Sqrt, floor, abs all compile, but fail to run. Others in the commented out code fail to compile.

I am traveling until mid-next week so won’t be able to look until then; appreciate your taking the time to get this running on another platform. Can only help the overall effort.

Regards,

- Steve

[Steven Núñez]

We do need compound data. It’s a fundamental xlispstat idea that, although loosely defined, is pervasive in the core code that you’re looking at. We don’t need compound objects however; that never really seemed to be implemented in xlispstat and none of the functions in the xlispstat code use it.

Tierny has a description of compound data in his book, which you might be able to find online. Unfortunately  it isn’t really defined that well. As nearly as I can tell, isn’t analogous to common lisp sequences. From an implementation perspective, xlispstat defined it as anything that starts with a cons cell or is an array.

The way I’ve come to think about compound data as an internal structure is a list-of-list. Likely this is intended for processing the xlispstat equivalent of data frames. Perhaps Tony or someone that is familiar with xlispstat can comment. Those map-element* functions are also widely used. I ported a simple xlispstat function to run on the code I had, and investigated several others, and that was a common pattern in what I saw.

BTW: xlispstat will run, unmodified, on a Mac. I found going through the code very insightful. It’s also a great way to see how things are supposed to work.

- Steve

[David Hodge]

Hi

(sqrt '(1 2 3))
In cmpndp with (1 2 3) In cmpndp with (1 2 3) In cmpndp with (1 2 3) In cmpndp with (1
                                                                                     2
                                                                                     3)
(1.0f0 1.4142135f0 1.7320508f0)


when I try to do SIN, for instance, it bombs in an infinite loop.

I note the presence of circular lists and a comment about FASTMAP being able to handle that - not sure if thats the source of the trouble.

Steven Núñez

8 January 2015 5:54 pm

--
You received this message because you are subscribed to the Google Groups "Common Lisp Statistics" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lisp-stat+...@googlegroups.com.
To post to this group, send email to lisp...@googlegroups.com.
Visit this group at http://groups.google.com/group/lisp-stat.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Common Lisp Statistics" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lisp-stat+...@googlegroups.com.
To post to this group, send email to lisp...@googlegroups.com.
Visit this group at http://groups.google.com/group/lisp-stat.
For more options, visit https://groups.google.com/d/optout.

David Hodge

8 January 2015 5:12 pm

[A.J. Rossini]

Am hoping for saturday to continue, no success before -- Steve, have you read the notes and comments in the CLS src/data  files?  If I recall, I wrote a critique of XLS's flaws (as per the second generation of data set storage, ie R's dataframe). 

Of course for some applications we are back to the XLS column store which is what RHO is about.  

For the math functions, there is some mystical voodoo happening to support that syntax -- why not just macro-ize a suitable MAP function?  There is no pressing need to make the syntax work, you are saving an hour of brain work in exchange for a week of hard design.   

Premature optimization is the root of all evil. 

 Also, LISTOFLISTS has code (combined with XREF) for basic data management.  Basic :).  

For random access, we want arrays or similar, not lists. 

Not for medium data.  

Best,

-tony

Sent from }*~£%¥>%=,?

[David Hodge]

Hi TOny,

I would certainly agree that there is voodoo.

And from reading the code a bit, its not that clear that its really worth the effort, as Tony also notes.

Steve, what are you trying to do ,  possibly things out of LISP-MATRIX will help?

Cheers

A.J. Rossini

8 January 2015 8:02 pm

--
You received this message because you are subscribed to the Google Groups "Common Lisp Statistics" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lisp-stat+...@googlegroups.com.
To post to this group, send email to lisp...@googlegroups.com.
Visit this group at http://groups.google.com/group/lisp-stat.
For more options, visit https://groups.google.com/d/optout.

David Hodge

8 January 2015 6:21 pm

[A.J. Rossini]

There is another package that supports generic vectorization of math functions, David, I think I found it from some comments you made?  Cl-ana had it maybe? 

But GSLL might be what you are looking for Steve? 

[Steven Núñez]

Hi Tony,

There’s a couple of threads in your message:

I read the notes. I’m not looking at data frames, just vectorized math,  but it’s got to be robust and, hopefully, help get cl-stat work on CCL, which is really where I need to because of the objc bridge. This started because of a generic need for vectorized math on common lisp. Does RHO support vectorized math?

CL-ana isn’t an option because it’s GPL infected.

Happy to take the route of least resistance, but I suspect that in the end any robust set of math macros is going to duplicate what cl-stat has already done. Besides, getting it running and having a close look at what the bug is here will only make cl-stat better. We’ve identified the problem, an infinite loop in map-elements*; question is: why?

At the end of the day, I’d like to kill two birds with one stone: give me natural syntax vectored math and get CL-stat on one more platform. Besides, understanding that syntax voodoo will certainly help me understand both CL-stat and mystical LISP chants.

Regards,

- Steve

[David Hodge]

Hi Steve,

All of the operators work in CLS. (eg sin, cos, tan, floor etc)

If you just install cls from quicklisp (assuming you have gsll installed first) you should be able to verify this pretty easily.

At the moment its pointing to something missing from the clsm repo, but I have not grokked the code enough to know what.

I would say that using this vectorised math syntax for anything other than small datasets is likely to be disappointing - there is a lot of overhead in the code, for not much value.

I am poking around a bit and its fun, I have to say. I will let you know if I find anything useful.

Steven Núñez

9 January 2015 3:14 am

A.J. Rossini

8 January 2015 10:27 pm

There is another package that supports generic vectorization of math functions, David, I think I found it from some comments you made?  Cl-ana had it maybe? 

But GSLL might be what you are looking for Steve? 

On Thursday, January 8, 2015, David Hodge <david...@gmail.com> wrote:

--
Sent from }*~£%¥>%=,?
--
You received this message because you are subscribed to the Google Groups "Common Lisp Statistics" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lisp-stat+...@googlegroups.com.
To post to this group, send email to lisp...@googlegroups.com.
Visit this group at http://groups.google.com/group/lisp-stat.
For more options, visit https://groups.google.com/d/optout.

David Hodge

8 January 2015 10:24 pm

[Steven Núñez]

Hi Dave,

Interesting. I’m going to have to try the quick lisp version when I get back.

Any reason that you can see for the inefficiencies in the vector math? It would be interesting to compare the performance on some medium sized arrays with that of xlispstat, which had an underlying implementation in C.

I suppose though, in the context of CL-stat going forward, how are we going to handle vector wise mathematics on large arrays and lists? Forcing everything to use one of the matrix libraries seems a bit inconvenient, not to mention introducing an unnatural syntax.

- Steve

[David Hodge]

Steven Núñez wrote:
> Any reason that you can see for the inefficiencies in the vector math?
> It would be interesting to compare the performance on some medium
> sized arrays with that of xlispstat, which had an underlying
> implementation in C.

The excessive checking for compund data types. Granted, the intention of
the syntax is to aid repl based exploring, but for larger data sets you
are paying a lot in IMHO unneccessary overhead. I'll do some timings
later and see if my assertion is backed up or not. This leads nicely
into your next para

>
> I suppose though, in the context of CL-stat going forward, how are we
> going to handle vector wise mathematics on large arrays and lists?
> Forcing everything to use one of the matrix libraries seems a bit
> inconvenient, not to mention introducing an unnatural syntax.

its all down to use cases in a way. I am most interested in climatology
timeseries data.For me, there is a development phase where i am
inspecting the data for interesting features, outliers, breakpoints,
quality control flags, non climactic changes (eg instrument failure or
changes to the measurement methodology) and so on and then validating
model choice, parameters etc. so its descriptive statistics and
summaries mainly. Infrequently, I might want to do something more
detailed (eg some sort of spectral analysis) , but I tend to write a
function for that, especially if its over a large dataset as I am often
not sure of the exact question and such exercises typically go through
several iterations.

Now, I have been thinking about dataframes to handle this. I have come
across quite a nice query language , so its easy to do subsetting and
then apply whatever processing across the rows. I need row based access
generally as attributes in the row drive certain data transformations
(eg the station was moved on this date, so I have to adjust for that
specific record, stuff like that). I use group-by for sub
totals/averages etc when i need them. for the large datasets, vectorised
operations don't make much sense - for the subsets they can do. The
other thing I need to figure out is cases of missing data and quality
controlled data - missing might be because the station has not reported,
or it might be that the instrument failed - usually I stop the
calculation if i don't see a report, but I might choose to infill due to
instrument failure. It would be nice to classify these things easily
upon ingestion and have the various operations deal with them. Writing
an efficient vectorised operation

Picking up some subset of a dataframe and handing it off to BLAS or FFTW
is pretty straightforward - LISP-MATRIX shows us how (and in fact gives
us a pretty nice api as well).

Its a good thing that you asked the original question Steve, its caused
me to read Tony's latest files re ModelFrames etc and its got me
thinking. I must say

Cheers

[David Hodge]

Hi Steve,

David Hodge

10 January 2015 12:27 pm

Steven Núñez wrote:

Any reason that you can see for the inefficiencies in the vector math? It would be interesting to compare the performance on some medium sized arrays with that of xlispstat, which had an underlying implementation in C.

The excessive checking for compund data types. Granted, the intention of the syntax is to aid repl based exploring, but for larger data sets you are paying a lot in IMHO unneccessary overhead. I'll do some timings later and see if my assertion is backed up or not. This leads nicely into your next para

Depending on the operator the vectorised operators are anywhere between 2 to 10 times slower. So trig operations (sin, cos, tan) are 2-3 times slower, addition about 6 times slower and float about 10.

In interactive use, with small data, its not noticable - but will quickly mount up for serious work I think. And there is a boatload of consing as well, especially as the amount of data increases.

Cheers

I suppose though, in the context of CL-stat going forward, how are we going to handle vector wise mathematics on large arrays and lists? Forcing everything to use one of the matrix libraries seems a bit inconvenient, not to mention introducing an unnatural syntax.

its all down to use cases in a way. I am most interested in climatology timeseries data.For me, there is a development phase where i am inspecting the data for interesting features, outliers, breakpoints, quality control flags, non climactic changes (eg instrument failure or changes to the measurement methodology)  and so on and then validating model choice, parameters etc. so its descriptive statistics and summaries mainly. Infrequently, I might want to do something more detailed (eg some sort of spectral analysis)   , but I tend to write a function for that, especially if its over a large dataset as I am often not sure of the exact question and such exercises typically go through several iterations.

Now, I have been thinking about dataframes to handle this. I have come across quite a nice query language  , so its easy to do subsetting and then apply whatever processing across the rows. I need row based access generally as attributes in the row drive certain data transformations (eg the station was moved on this date, so I have to adjust for that specific record, stuff like that).   I use group-by for sub totals/averages etc when i need them. for the large datasets, vectorised operations don't make much sense - for the subsets they can do. The other thing I need to figure out is cases of missing data and quality controlled data - missing might be because the station has not reported, or it might be that the instrument failed - usually I stop the calculation if i don't see a report, but I might choose to infill due to instrument failure. It would be nice to classify these things easily upon ingestion and have the various operations deal with them. Writing an efficient vectorised operation

Picking up some subset of a dataframe and handing it off to BLAS or FFTW is pretty straightforward - LISP-MATRIX shows us how (and in fact gives us a pretty nice api as well).

Its a good thing that you asked the original question Steve, its caused me to read Tony's latest files re ModelFrames etc  and its got me thinking. I must say

Cheers

Steven Núñez

10 January 2015 8:59 am

Hi Dave,

Interesting. I’m going to have to try the quick lisp version when I get back.

Any reason that you can see for the inefficiencies in the vector math? It would be interesting to compare the performance on some medium sized arrays with that of xlispstat, which had an underlying implementation in C.

I suppose though, in the context of CL-stat going forward, how are we going to handle vector wise mathematics on large arrays and lists? Forcing everything to use one of the matrix libraries seems a bit inconvenient, not to mention introducing an unnatural syntax.

- Steve

From: David Hodge <david...@gmail.com>
Reply-To: "lisp...@googlegroups.com" <lisp...@googlegroups.com>
Date: Thursday, 8 January 2015 19:41
To: "lisp...@googlegroups.com" <lisp...@googlegroups.com>
Subject: Re: [lisp-stat] Element-wise Math in CL, part 2

Hi Steve,

All of the operators work in CLS. (eg sin, cos, tan, floor etc)

If you just install cls from quicklisp (assuming you have gsll installed first) you should be able to verify this pretty easily.

At the moment its pointing to something missing from the clsm repo, but I have not grokked the code enough to know what.

I would say that using this vectorised math syntax for anything other than small datasets is likely to be disappointing - there is a lot of overhead in the code, for not much value.

I am poking around a bit and its fun, I have to say. I will let you know if I find anything useful.

--
You received this message because you are subscribed to the Google Groups "Common Lisp Statistics" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lisp-stat+...@googlegroups.com.
To post to this group, send email to lisp...@googlegroups.com.
Visit this group at http://groups.google.com/group/lisp-stat.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Common Lisp Statistics" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lisp-stat+...@googlegroups.com.
To post to this group, send email to lisp...@googlegroups.com.
Visit this group at http://groups.google.com/group/lisp-stat.
For more options, visit https://groups.google.com/d/optout.

David Hodge

9 January 2015 4:41 pm

[Steven Núñez]

[Steven Núñez]

Interesting. Given that most all data analysis is going to use vectors, what’s the CLS strategy here? Develop something that does work, or use a third partly library?

[David Hodge]

Well, the stuff we have been talking about is, to my mind, a repl convenience - not really something one would use in heavy weight applications.


I have no doubt it could be improved performance wise if one really wished - just get rid of the excessive compound data checking for starters. As I am not really sure what compound data actually is, or what one would use it for , i can only note the performance hits that it makes. I should note that my tests are done with default optimisation settings, I could compile things with higher speed to see if it makes a difference.

The longer term strategy, as Tony notes elsewhere would be dataframes/Modelframes using a column store and providing a suite of functions on that.

I suppose a good question would be where does CLS fit in the analytic workflow - is it a swiss army knife that does data reduction as well as analysis or does it focus on the analysis domain only

Steven Núñez

16 January 2015 1:06 pm

David Hodge

12 January 2015 10:10 pm

[Steven Núñez]

Compound data is just a list or an array. I don’t know that I’d call it just a REPL convenience. There’s lot’s of reasons I’d want to do vector wise math in production scripts that are run regularly.

I was hoping Tony would chime in here with an update on RHO. I’ve seen signs of recent activity on the github repository. In my mind though the data frame is independent of the vector-wise math, since I’m still very likely to do vector operations on the rows and columns of a data frame, and it wouldn’t make sense to have two mechanisms for that (one for REPL and one for data frames). We need an underlying efficient implementation of vector math no matter which way we go; the more natural the syntax the better.

As far as where it fits in, I see it as a replacement for R and the python suite, addressing the large-memory weaknesses in R pointed about in the ‘back to the future’ paper. The fact that LISP could be used in production gives it a great advantage, though realistically unless some community evolves around CL-stat and advanced (distributed, IMO) analytics we won’t have a chance at catching up.

Perhaps the best bet would be to implement a common lisp DSL on Spark and admit defeat as an execution engine but at least be able to work in a more efficient syntax. Thinking about it, we might slowly be able to extend down the stack from there at some point, but the sad fact is there there is no way to do analytics with LISP at all. I’d be happy with any solution that provides that.

- Steve

[David Hodge]

well, you might want to look at Tamas Papp cl-num-utils. certainly on github, probably in quicklisp, though i have not looked.

This has a Boost license and probably does some of what you need for elementwise operations. At least would provide a good place to start

cheers

Steven Núñez

16 January 2015 3:23 pm

Compound data is just a list or an array. I don’t know that I’d call it just a REPL convenience. There’s lot’s of reasons I’d want to do vector wise math in production scripts that are run regularly.

I was hoping Tony would chime in here with an update on RHO. I’ve seen signs of recent activity on the github repository. In my mind though the data frame is independent of the vector-wise math, since I’m still very likely to do vector operations on the rows and columns of a data frame, and it wouldn’t make sense to have two mechanisms for that (one for REPL and one for data frames). We need an underlying efficient implementation of vector math no matter which way we go; the more natural the syntax the better.

As far as where it fits in, I see it as a replacement for R and the python suite, addressing the large-memory weaknesses in R pointed about in the ‘back to the future’ paper. The fact that LISP could be used in production gives it a great advantage, though realistically unless some community evolves around CL-stat and advanced (distributed, IMO) analytics we won’t have a chance at catching up.

Perhaps the best bet would be to implement a common lisp DSL on Spark and admit defeat as an execution engine but at least be able to work in a more efficient syntax. Thinking about it, we might slowly be able to extend down the stack from there at some point, but the sad fact is there there is no way to do analytics with LISP at all. I’d be happy with any solution that provides that.

- Steve

From: David Hodge <david...@gmail.com>
Reply-To: "lisp...@googlegroups.com" <lisp...@googlegroups.com>
Date: Thursday, 15 January 2015 16:58
To: "lisp...@googlegroups.com" <lisp...@googlegroups.com>
Subject: Re: [lisp-stat] Element-wise Math in CL, part 2

Well, the stuff we have been talking about is, to my mind, a repl convenience - not really something one would use in heavy weight applications.


I have no doubt it could be improved performance wise if one really wished - just get rid of the excessive compound data checking for starters. As I am not really sure what compound data actually is, or what one would use it for , i can only note the performance hits that it makes. I should note that my tests are done with default optimisation settings, I could compile things with higher speed to see if it makes a difference.

The longer term strategy, as Tony notes elsewhere would be dataframes/Modelframes using a column store and providing a suite of functions on that.

I suppose a good question would be where does CLS fit in the analytic workflow - is it a swiss army knife that does data reduction as well as analysis or does it focus on the analysis domain only

--
You received this message because you are subscribed to the Google Groups "Common Lisp Statistics" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lisp-stat+...@googlegroups.com.
To post to this group, send email to lisp...@googlegroups.com.
Visit this group at http://groups.google.com/group/lisp-stat.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Common Lisp Statistics" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lisp-stat+...@googlegroups.com.
To post to this group, send email to lisp...@googlegroups.com.
Visit this group at http://groups.google.com/group/lisp-stat.
For more options, visit https://groups.google.com/d/optout.

David Hodge

16 January 2015 1:58 pm

[Tamas Papp]

The CLNU library works, but it far from perfect and is unlikely to be
maintained further by me. I am 99% sure that I will be transitioning to
Julia in the medium run.

Honestly, I don't see much of a future for Lisp-based numerical
work. Lisp is extremely powerful, but Common Lisp has quite a few
limitations that mean that you have to pick 2 out of (1) fast, (2)
convenient, and (3) portable if you want to write numerical code. The
two biggest limitations for me are lack of parametric types and limited
support for arrays with a specialized element type. Both of these could
be overcome, but that would mean redesigning the language, or a very
heavy layer on CL that would need to involve the compilers, too.

I don't mean to discourage anyone though, I just wanted to make it clear
why my CL libraries are unlikely to get any extra work.

Best,

Tamas

On Fri, Jan 16 2015, David Hodge <david...@gmail.com> wrote:

> well, you might want to look at Tamas Papp cl-num-utils. certainly on
> github, probably in quicklisp, though i have not looked.
>
> This has a Boost license and probably does some of what you need for
> elementwise operations. At least would provide a good place to start
>
> cheers
>

>> Steven Núñez <mailto:steven...@illation.com>

>> 16 January 2015 3:23 pm
>> Compound data is just a list or an array. I don’t know that I’d call
>> it just a REPL convenience. There’s lot’s of reasons I’d want to do
>> vector wise math in production scripts that are run regularly.
>>
>> I was hoping Tony would chime in here with an update on RHO. I’ve seen
>> signs of recent activity on the github repository. In my mind though
>> the data frame is independent of the vector-wise math, since I’m still
>> very likely to do vector operations on the rows and columns of a data
>> frame, and it wouldn’t make sense to have two mechanisms for that (one
>> for REPL and one for data frames). We need an underlying efficient
>> implementation of vector math no matter which way we go; the more
>> natural the syntax the better.
>>
>> As far as where it fits in, I see it as a replacement for R and the
>> python suite, addressing the large-memory weaknesses in R pointed
>> about in the ‘back to the future’ paper. The fact that LISP could be
>> used in production gives it a great advantage, though realistically
>> unless some community evolves around CL-stat and advanced
>> (distributed, IMO) analytics we won’t have a chance at catching up.
>>
>> Perhaps the best bet would be to implement a common lisp DSL on Spark
>> and admit defeat as an execution engine but at least be able to work
>> in a more efficient syntax. Thinking about it, we might slowly be able
>> to extend down the stack from there at some point, but the sad fact is

>> there there is /no/ way to do analytics with LISP at all. I’d be happy

>> <mailto:lisp-stat+...@googlegroups.com>.

>> To post to this group, send email to lisp...@googlegroups.com

>> <mailto:lisp...@googlegroups.com>.

>> Visit this group at http://groups.google.com/group/lisp-stat.
>> For more options, visit https://groups.google.com/d/optout.
>> --
>> You received this message because you are subscribed to the Google
>> Groups "Common Lisp Statistics" group.
>> To unsubscribe from this group and stop receiving emails from it, send
>> an email to lisp-stat+...@googlegroups.com

>> <mailto:lisp-stat+...@googlegroups.com>.

>> To post to this group, send email to lisp...@googlegroups.com

>> <mailto:lisp...@googlegroups.com>.

>> Visit this group at http://groups.google.com/group/lisp-stat.
>> For more options, visit https://groups.google.com/d/optout.

>> David Hodge <mailto:david...@gmail.com>

>> 16 January 2015 1:58 pm
>> Well, the stuff we have been talking about is, to my mind, a repl
>> convenience - not really something one would use in heavy weight
>> applications.
>>
>>
>> I have no doubt it could be improved performance wise if one really
>> wished - just get rid of the excessive compound data checking for
>> starters. As I am not really sure what compound data actually is, or
>> what one would use it for , i can only note the performance hits that
>> it makes. I should note that my tests are done with default
>> optimisation settings, I could compile things with higher speed to see
>> if it makes a difference.
>>
>> The longer term strategy, as Tony notes elsewhere would be
>> dataframes/Modelframes using a column store and providing a suite of
>> functions on that.
>>
>> I suppose a good question would be where does CLS fit in the analytic
>> workflow - is it a swiss army knife that does data reduction as well
>> as analysis or does it focus on the analysis domain only
>>
>>

>> Steven Núñez <mailto:steven...@illation.com>

>> 16 January 2015 1:06 pm
>>
>> Interesting. Given that most all data analysis is going to use
>> vectors, what’s the CLS strategy here? Develop something that does
>> work, or use a third partly library?
>>

>> *From:*lisp...@googlegroups.com [mailto:lisp...@googlegroups.com]
>> *On Behalf Of *David Hodge
>> *Sent:* Monday, 12 January 2015 1:11
>> *To:* lisp...@googlegroups.com
>> *Subject:* Re: [lisp-stat] Element-wise Math in CL, part 2
>>
>> Hi Steve,
>>
>>
>> *David Hodge* <mailto:david...@gmail.com>

>> *Steven Núñez* <mailto:steven...@illation.com>

>>
>> 10 January 2015 8:59 am
>>
>> Hi Dave,
>>
>> Interesting. I’m going to have to try the quick lisp version when
>> I get back.
>>
>> Any reason that you can see for the inefficiencies in the vector
>> math? It would be interesting to compare the performance on some
>> medium sized arrays with that of xlispstat, which had an
>> underlying implementation in C.
>>
>> I suppose though, in the context of CL-stat going forward, how are
>> we going to handle vector wise mathematics on large arrays and
>> lists? Forcing everything to use one of the matrix libraries seems
>> a bit inconvenient, not to mention introducing an unnatural syntax.
>>
>> - Steve
>>

>> *From: *David Hodge <david...@gmail.com
>> <mailto:david...@gmail.com>>
>> *Reply-To: *"lisp...@googlegroups.com
>> <mailto:lisp...@googlegroups.com>" <lisp...@googlegroups.com
>> <mailto:lisp...@googlegroups.com>>
>> *Date: *Thursday, 8 January 2015 19:41
>> *To: *"lisp...@googlegroups.com
>> <mailto:lisp...@googlegroups.com>" <lisp...@googlegroups.com
>> <mailto:lisp...@googlegroups.com>>
>> *Subject: *Re: [lisp-stat] Element-wise Math in CL, part 2

>>
>> Hi Steve,
>>
>> All of the operators work in CLS. (eg sin, cos, tan, floor etc)
>>
>> If you just install cls from quicklisp (assuming you have gsll
>> installed first) you should be able to verify this pretty easily.
>>
>> At the moment its pointing to something missing from the clsm
>> repo, but I have not grokked the code enough to know what.
>>
>> I would say that using this vectorised math syntax for anything
>> other than small datasets is likely to be disappointing - there is
>> a lot of overhead in the code, for not much value.
>>
>> I am poking around a bit and its fun, I have to say. I will let
>> you know if I find anything useful.
>>
>> --
>> You received this message because you are subscribed to the Google
>> Groups "Common Lisp Statistics" group.
>> To unsubscribe from this group and stop receiving emails from it,
>> send an email to lisp-stat+...@googlegroups.com

>> <mailto:lisp-stat+...@googlegroups.com>.

>> To post to this group, send email to lisp...@googlegroups.com

>> <mailto:lisp...@googlegroups.com>.

>> Visit this group at http://groups.google.com/group/lisp-stat.
>> For more options, visit https://groups.google.com/d/optout.
>>
>> --
>> You received this message because you are subscribed to the Google
>> Groups "Common Lisp Statistics" group.
>> To unsubscribe from this group and stop receiving emails from it,
>> send an email to lisp-stat+...@googlegroups.com

>> <mailto:lisp-stat+...@googlegroups.com>.

>> To post to this group, send email to lisp...@googlegroups.com

>> <mailto:lisp...@googlegroups.com>.

>> Visit this group at http://groups.google.com/group/lisp-stat.
>> For more options, visit https://groups.google.com/d/optout.
>>

>> *David Hodge* <mailto:david...@gmail.com>

>>
>> 9 January 2015 4:41 pm
>>
>> Hi Steve,
>>
>> All of the operators work in CLS. (eg sin, cos, tan, floor etc)
>>
>> If you just install cls from quicklisp (assuming you have gsll
>> installed first) you should be able to verify this pretty easily.
>>
>> At the moment its pointing to something missing from the clsm
>> repo, but I have not grokked the code enough to know what.
>>
>> I would say that using this vectorised math syntax for anything
>> other than small datasets is likely to be disappointing - there is
>> a lot of overhead in the code, for not much value.
>>
>> I am poking around a bit and its fun, I have to say. I will let
>> you know if I find anything useful.
>>

>> *Steven Núñez* <mailto:steven...@illation.com>

>>
>> 9 January 2015 3:14 am
>>
>> Hi Tony,
>>
>> There’s a couple of threads in your message:
>>
>> I read the notes. I’m not looking at data frames, just vectorized
>> math, but it’s got to be robust and, hopefully, help get cl-stat
>> work on CCL, which is really where I need to because of the objc
>> bridge. This started because of a generic need for vectorized math
>> on common lisp. Does RHO support vectorized math?
>>
>> CL-ana isn’t an option because it’s GPL infected.
>>
>> Happy to take the route of least resistance, but I suspect that in
>> the end any robust set of math macros is going to duplicate what
>> cl-stat has already done. Besides, getting it running and having a
>> close look at what the bug is here will only make cl-stat better.
>> We’ve identified the problem, an infinite loop in map-elements*;
>> question is: why?
>>
>> At the end of the day, I’d like to kill two birds with one stone:
>> give me natural syntax vectored math and get CL-stat on one more
>> platform. Besides, understanding that syntax voodoo will certainly
>> help me understand both CL-stat and mystical LISP chants.
>>
>> Regards,
>>
>> - Steve
>>

>> *From: *"A.J. Rossini" <blind...@gmail.com
>> <mailto:blind...@gmail.com>>
>> *Reply-To: *"lisp...@googlegroups.com
>> <mailto:lisp...@googlegroups.com>" <lisp...@googlegroups.com
>> <mailto:lisp...@googlegroups.com>>
>> *Date: *Thursday, 8 January 2015 1:27
>> *To: *"lisp...@googlegroups.com
>> <mailto:lisp...@googlegroups.com>" <lisp...@googlegroups.com
>> <mailto:lisp...@googlegroups.com>>
>> *Subject: *Re: [lisp-stat] Element-wise Math in CL, part 2

>>
>> There is another package that supports generic vectorization of
>> math functions, David, I think I found it from some comments you
>> made? Cl-ana had it maybe?
>>
>> But GSLL might be what you are looking for Steve?
>>
>> On Thursday, January 8, 2015, David Hodge <david...@gmail.com

>> <mailto:david...@gmail.com>> wrote:
>>
>>
>>
>> --
>> Sent from }*~£%¥>%=,?
>> --
>> You received this message because you are subscribed to the Google
>> Groups "Common Lisp Statistics" group.
>> To unsubscribe from this group and stop receiving emails from it,
>> send an email to lisp-stat+...@googlegroups.com

>> <mailto:lisp-stat+...@googlegroups.com>.

>> To post to this group, send email to lisp...@googlegroups.com

>> <mailto:lisp...@googlegroups.com>.

>> Visit this group at http://groups.google.com/group/lisp-stat.
>> For more options, visit https://groups.google.com/d/optout.
>>
>> --
>> You received this message because you are subscribed to the Google
>> Groups "Common Lisp Statistics" group.
>> To unsubscribe from this group and stop receiving emails from it,
>> send an email to lisp-stat+...@googlegroups.com

>> <mailto:lisp-stat+...@googlegroups.com>.

>> To post to this group, send email to lisp...@googlegroups.com

>> <mailto:lisp...@googlegroups.com>.

>> Visit this group at http://groups.google.com/group/lisp-stat.
>> For more options, visit https://groups.google.com/d/optout.
>>

>> *A.J. Rossini* <mailto:blind...@gmail.com>

>>
>> 8 January 2015 10:27 pm
>>
>> There is another package that supports generic vectorization of
>> math functions, David, I think I found it from some comments you
>> made? Cl-ana had it maybe?
>>
>> But GSLL might be what you are looking for Steve?
>>
>> On Thursday, January 8, 2015, David Hodge <david...@gmail.com

>> <mailto:david...@gmail.com>> wrote:
>>
>>
>>
>> --
>> Sent from }*~£%¥>%=,?
>> --
>> You received this message because you are subscribed to the Google
>> Groups "Common Lisp Statistics" group.
>> To unsubscribe from this group and stop receiving emails from it,
>> send an email to lisp-stat+...@googlegroups.com

>> <mailto:lisp-stat+...@googlegroups.com>.

>> To post to this group, send email to lisp...@googlegroups.com

>> <mailto:lisp...@googlegroups.com>.

>> Visit this group at http://groups.google.com/group/lisp-stat.
>> For more options, visit https://groups.google.com/d/optout.
>>
>> --
>> You received this message because you are subscribed to the Google
>> Groups "Common Lisp Statistics" group.
>> To unsubscribe from this group and stop receiving emails from it, send
>> an email to lisp-stat+...@googlegroups.com

>> <mailto:lisp-stat+...@googlegroups.com>.

>> To post to this group, send email to lisp...@googlegroups.com

>> <mailto:lisp...@googlegroups.com>.

>> Visit this group at http://groups.google.com/group/lisp-stat.
>> For more options, visit https://groups.google.com/d/optout.
>>
>> --
>> You received this message because you are subscribed to the Google
>> Groups "Common Lisp Statistics" group.
>> To unsubscribe from this group and stop receiving emails from it, send
>> an email to lisp-stat+...@googlegroups.com

>> <mailto:lisp-stat+...@googlegroups.com>.

>> To post to this group, send email to lisp...@googlegroups.com

>> <mailto:lisp...@googlegroups.com>.

>> Visit this group at http://groups.google.com/group/lisp-stat.
>> For more options, visit https://groups.google.com/d/optout.

>> David Hodge <mailto:david...@gmail.com>

>> 12 January 2015 10:10 pm
>> Hi Steve,
>>

>>> David Hodge <mailto:david...@gmail.com>

>>> Steven Núñez <mailto:steven...@illation.com>

>>> <mailto:lisp-stat+...@googlegroups.com>.

>>> To post to this group, send email to lisp...@googlegroups.com

>>> <mailto:lisp...@googlegroups.com>.

>>> Visit this group at http://groups.google.com/group/lisp-stat.
>>> For more options, visit https://groups.google.com/d/optout.
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Common Lisp Statistics" group.
>>> To unsubscribe from this group and stop receiving emails from it,
>>> send an email to lisp-stat+...@googlegroups.com

>>> <mailto:lisp-stat+...@googlegroups.com>.

>>> To post to this group, send email to lisp...@googlegroups.com

>>> <mailto:lisp...@googlegroups.com>.

>>> Visit this group at http://groups.google.com/group/lisp-stat.
>>> For more options, visit https://groups.google.com/d/optout.

>>> David Hodge <mailto:david...@gmail.com>

>>> 9 January 2015 4:41 pm
>>> Hi Steve,
>>>
>>> All of the operators work in CLS. (eg sin, cos, tan, floor etc)
>>>
>>> If you just install cls from quicklisp (assuming you have gsll
>>> installed first) you should be able to verify this pretty easily.
>>>
>>> At the moment its pointing to something missing from the clsm repo,
>>> but I have not grokked the code enough to know what.
>>>
>>> I would say that using this vectorised math syntax for anything other
>>> than small datasets is likely to be disappointing - there is a lot of
>>> overhead in the code, for not much value.
>>>
>>> I am poking around a bit and its fun, I have to say. I will let you
>>> know if I find anything useful.
>>>

>>> Steven Núñez <mailto:steven...@illation.com>

>>> <mailto:david...@gmail.com>> wrote:
>>>
>>>
>>> --
>>> Sent from }*~£%¥>%=,?
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Common Lisp Statistics" group.
>>> To unsubscribe from this group and stop receiving emails from it,
>>> send an email to lisp-stat+...@googlegroups.com

>>> <mailto:lisp-stat+...@googlegroups.com>.

>>> To post to this group, send email to lisp...@googlegroups.com

>>> <mailto:lisp...@googlegroups.com>.

>>> Visit this group at http://groups.google.com/group/lisp-stat.
>>> For more options, visit https://groups.google.com/d/optout.
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Common Lisp Statistics" group.
>>> To unsubscribe from this group and stop receiving emails from it,
>>> send an email to lisp-stat+...@googlegroups.com

>>> <mailto:lisp-stat+...@googlegroups.com>.

>>> To post to this group, send email to lisp...@googlegroups.com

>>> <mailto:lisp...@googlegroups.com>.

>>> Visit this group at http://groups.google.com/group/lisp-stat.
>>> For more options, visit https://groups.google.com/d/optout.

>>> A.J. Rossini <mailto:blind...@gmail.com>

>>> 8 January 2015 10:27 pm
>>> There is another package that supports generic vectorization of math
>>> functions, David, I think I found it from some comments you made?
>>> Cl-ana had it maybe?
>>>
>>> But GSLL might be what you are looking for Steve?
>>>
>>> On Thursday, January 8, 2015, David Hodge <david...@gmail.com

>>> <mailto:david...@gmail.com>> wrote:
>>>
>>>
>>> --
>>> Sent from }*~£%¥>%=,?
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Common Lisp Statistics" group.
>>> To unsubscribe from this group and stop receiving emails from it,
>>> send an email to lisp-stat+...@googlegroups.com

>>> <mailto:lisp-stat+...@googlegroups.com>.

>>> To post to this group, send email to lisp...@googlegroups.com

>>> <mailto:lisp...@googlegroups.com>.

>>> Visit this group at http://groups.google.com/group/lisp-stat.
>>> For more options, visit https://groups.google.com/d/optout.

>> David Hodge <mailto:david...@gmail.com>

[Steven Núñez]

Nice summary on some the problems with CL. Seems standardisation really did take the wind out of the sails.

I have known about Julia for some time but never looked into it. Impressive. Natural mathematical syntax, complies to native code, can use C libraries without fuss and has lisp style macros. Seems too good to be true.

Have you used it in practice? Does it live up to its claims? Where did the project come from? I can't find anything about its evolution.

Sent from my iPhone

> To unsubscribe from this group and stop receiving emails from it, send an email to lisp-stat+...@googlegroups.com.
> To post to this group, send email to lisp...@googlegroups.com.

[Tamas Papp]

Hi Steven,

I have only been using Julia for a couple of months. It does live up the
claims, but of course it is still a bit rough around the edges (being in
development, eg they just standardized docstrings, etc), and its
metaprogramming facilities are nowhere near as advanced as CL (that
said, I have used that stuff in CL very rarely, method combinations a
couple of times, and MOP never, but that may just reflect my lack of
experience). And, of course, ESS doesn't even come close to SLIME, and
ESS for Julia is particularly incomplete.

Regarding the history, see the papers here:
http://julialang.org/publications/

Best,

Tamas