I've been working on a contract whose project has just gone live. It's an important system for the company concerned and it has nothing to do with Lisp of course (it's the usual enterprise crapware).
Needless to say it fell over in the first few hours, the way these things do (people were making half-serious jokes about whether it would make it into The Register). It''s all back now but people would like to understand what went wrong obviously.
I spent a long time on Friday grovelling ineffectively over log files (really huge log files, simple awk scripts took tens of minutes to process them) to try and find bad things. Today I realised that there's a much smaller log file which contains just lists of API calls, when they started, how long they took and the PID of the process that handled them (which you can use to find the appropriate big log file to look in). I realised that, if I could make a picture of the service times that might be interesting (I can't do anything without pictures). So I did that, via my standard trick of a tiny awk (or perl) front-end to convert things from line-based into sexp-based, and then some Lisp code to spit out something gnuplot could print.
And indeed it was interesting. Generally the service times were a hundredth of a second or so, but there were some which were tens of *minutes*. So, some more playing based on the original code and I have a list of the offending services, times, and PIDs for things taking more than some number of times the median time. With suitable settings this give a few tens of calls which I can now go and look at the detailed logs for to work out what was going on.
None of this is special magic Lisp code: in fact there's really nothing remarkable about it at all. I could have done it in Perl. Was Lisp better than Perl? For me, yes, because I'm a better (less bad) Lisp programmer than Perl programmer and the environment I have for Lisp is better than the one I have for Perl. But significantly it wasn't *worse*: nothing was difficult, and nothing got in the way, it just all worked easily and well. It was just *ordinary*, and I'm coming to the view that that's what you want: a language which is just there and does what you want without some glamourous computational pyrotechnics.
Common Lisp actually turns out to be quite good at that.
> None of this is special magic Lisp code: in fact there's really nothing
> remarkable about it at all. I could have done it in Perl. Was Lisp
> better than Perl? For me, yes, because I'm a better (less bad) Lisp
> programmer than Perl programmer and the environment I have for Lisp is
> better than the one I have for Perl. But significantly it wasn't
> *worse*: nothing was difficult, and nothing got in the way, it just all
> worked easily and well. It was just *ordinary*, and I'm coming to the
> view that that's what you want: a language which is just there and does
> what you want without some glamourous computational pyrotechnics. Common
> Lisp actually turns out to be quite good at that.
What's the performance? Long ago, some friends brought me in to grovel some log files in Lisp because the Perl execution time was up to tens of minutes. Perhaps they were using naive algorithms (as one does when groveling) but the Lisp execution times were in the tens of seconds.
Paul Wallich <p...@panix.com> wrote:
> What's the performance? Long ago, some friends brought me in to grovel
> some log files in Lisp because the Perl execution time was up to tens of
> minutes. Perhaps they were using naive algorithms (as one does when
> groveling) but the Lisp execution times were in the tens of seconds.
A few seconds, but this is only a day's worth of timing data.
> in). I realised that, if I could make a picture of the service times
> that might be interesting (I can't do anything without pictures). So I
> did that, via my standard trick of a tiny awk (or perl) front-end to
> convert things from line-based into sexp-based, and then some Lisp code
> to spit out something gnuplot could print.
I agree. In my experience 'visualization' always was, and still is, the
method of choice. In a science lab for example one would be completely
lost without it. Ways and means to carry this out are without number.
If one likes doing things in Lisp, CLTL2 offers an extended example on
how to create PostScript code. It needs some minor adjustments, at
least with the Lisps I've tried then, but it might pay off, depending
on the problem given.
On 2012-02-14 14:28:48 +0000, Michael Moeller said:
> I agree. In my experience 'visualization' always was, and still is, the
> method of choice.
I used to think this, but I now think that it depends on who you are: there do seem to be genuinely non-visual people. For a very long time[*] I used to just assume that these people were some kind of alien or just not smart, but I think I was entirely wrong about that.
--tim
[*] Actually, not really a very long time by the standards of how old I now am, if you see what I mean.
> On 2012-02-14 14:28:48 +0000, Michael Moeller said:
>> I agree. In my experience 'visualization' always was, and still is, the
>> method of choice.
> I used to think this, but I now think that it depends on who you are:
> there do seem to be genuinely non-visual people. For a very long time[*]
> I used to just assume that these people were some kind of alien or just
> not smart, but I think I was entirely wrong about that.
> --tim
> [*] Actually, not really a very long time by the standards of how old I
> now am, if you see what I mean.
Genuinely non-visual? To me this is a developmental disorder during
infancy. Or did you mean 'unwilling' to mess about with graphics stuff?
>> I used to think this, but I now think that it depends on who you are:
>> there do seem to be genuinely non-visual people. For a very long time[*]
>> I used to just assume that these people were some kind of alien or just
>> not smart, but I think I was entirely wrong about that.
>> [*] Actually, not really a very long time by the standards of how old I
>> now am, if you see what I mean.
> Genuinely non-visual? To me this is a developmental disorder during
> infancy. Or did you mean 'unwilling' to mess about with graphics stuff?
Some people are good at handling speech and textual information, that is, they can just read it and it makes sense for them without intermediate representation in a visual form. I'd say it is a strength, not disorder.
> On 2012-02-14 23:29:54 +0000, Michael Moeller said:
>> enuinely non-visual? To me this is a developmental disorder during
>> infancy. Or did you mean 'unwilling' to mess about with graphics stuff?
> I meant "don't think in pictures".
I see. I asked this just to be sure. In my opinion it's best to
choose whatever method is most appropriate. Admittedly a matter
of taste in many cases. If a data set is small enough to get
evaluated by simply reading through it we have no problem at all.
Of course there are ways and means to handle data other than by
visualization. For this reason industrial strength statistics
applications offer both methods.
> On 2012-02-15 14:03:27 +0000, Michael Moeller said:
>> Admittedly a matter
>> of taste in many cases.
> Well, my point was that no, it doesn't seem to be. It seems instead to
> be a matter of differing capacities for various styles of thinking.
People are of course more or less visually oriented but normally one
deliberately changes the style of thinking according to the problem
given. Sooner or later anything else will probably raise attention.
At least mine. This may depend on how close you look at the phenomenon.
On 2012-02-15 17:05:30 +0000, Michael Moeller said:
> People are of course more or less visually oriented but normally one
> deliberately changes the style of thinking according to the problem
> given. Sooner or later anything else will probably raise attention.
> At least mine. This may depend on how close you look at the phenomenon.
I can't get a parse of the second part of this. However my point was really that the differences appear to be really quite significant. I'm a long way at the visual end of things, and I find, for instance, programming in an environment which does not have good support for visualising programs (which is not pretty colours in the source code) sufficiently unpleasant that I don't really bother to do it any more unless someone's paying me to. There aren't many such environments I know of: I suspect most programmers are not very visual because of that fact.
Anyway, I've probably said more than is useful here.
> On 2012-02-15 17:05:30 +0000, Michael Moeller said:
>> People are of course more or less visually oriented but normally one
>> deliberately changes the style of thinking according to the problem
>> given. Sooner or later anything else will probably raise attention.
>> At least mine. This may depend on how close you look at the phenomenon.
> I can't get a parse of the second part of this.However my point was
> really that the differences appear to be really quite significant. I'm a
> long way at the visual end of things, and I find, for instance,
> programming in an environment which does not have good support for
> visualising programs (which is not pretty colours in the source code)
> sufficiently unpleasant that I don't really bother to do it any more
> unless someone's paying me to. There aren't many such environments I
> know of: I suspect most programmers are not very visual because of that
> fact.
> Anyway, I've probably said more than is useful here.
That's a new aspect. I don't see any connection between writing code and visualization. Anyway, in an environment such as industrial research and engineering a rather non-visual person will have a very
hard time.
Tim Bradshaw <t...@tfeb.org> writes:
> big log file to look in). I realised that, if I could make a picture
> of the service times that might be interesting (I can't do anything
