Oh, and does anyone know whether Alegro LISP would work
under Linux or would I need a different version?

Cheers

One problem with neural networks is that they're inherenly both
massively parallel and quite analog.  Doing them efficiently on a
mostly-sequential, definitely-digital computer can be a pain, even
where possible.

Then again, analog VLSI, while efficient for these computations, has
quite a long develop-compile-test cycle :-)

--
Arvid

Cheers

http://www.franz.com/support/docs/5.0/doc/cl/profiling.htm

Once you know where the bottlenecks are then you can attack with
optimizations to the algorithms and/or declarations.

--
John Watton
Alcoa Inc.

  well, you _could_ use adjustable vectors with fill pointers if such is
  your concern.  Common Lisp knows that the tradeoff that people made with
  lists carried a performance penalty.  look at VECTOR-PUSH and VECTOR-POP
  for ways to deal with unknown sizes and still get O(1) access times.

have you compiled your funtions ? if you don't it really slows things down...

Cheers

Apart from that, one of the most common pifalls of Lisp vs C
comparisons is that the data structures used in C are different from
those used in Lisp.  If I use a 'malloc-ed' array in C, I am bound to
get better performance than a list in Lisp, especially if you access
the i-th element out of an ordered traversal.

I bet that using lists with C++/STL would result in larger code and
slower as well w.r.t. an equivalent C/C++ array based implementation
(not to speak of any comparison with Lisp).

Cheers

CMUCL should run this kind of code really well (albeit I haven't even
tried compiling it other than the Y function so it might be buggy and
/ or need better declarations.

;; Code for a list based Reccurent Neural Network
;; We will need a global list of length = no_of_nodes, which gives the current activation levels of
;; every node in the network. This list will be updated and accessed by future functions.

;(defun setup_activity (no_of_nodes)
;   (setf *structure* nil)
;   (setf *structure* (cons 1 (dotimes (x no_of_nodes *structure*)
;                              (setf *structure* (cons '0 *structure*))))))

(defun make-activity (n)
  (let ((a (make-array n
                       :element-type 'single-float
                       :initial-element 0.0s0)))
    (setf (aref a 0) 1.0s0)
    a))

;;(setup_activity 10)
;;>(1 0 0 0 0 0 0 0 0 0 0)
;;(setup_activity 20)
;;>(1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0)

;; Starting with the functions to create a single neurone/node
;; I need a to create a structure to hold the weights and any other information about the node.

;(defstruct node
;  weights_list  ;a list of weights, including the bias weight
;  input_list)   ;a list of input locations pointing to positions in *structure* including the bias node

(defstruct (node
            (:constructor %make-node))
  weights-array
  input-array)

(defun make-node (weights-list input-list)
  (%make-node :weights-array (make-array (length weights-list)
                                         :element-type 'single-float
                                         :initial-contents weights-list)
              :input-array (make-array (length input-list)
                                       :element-type 'fixnum
                                       :initial-contents input-list)))

;(defun mk_node_struct (weights_list input_list)
;   (setf *node* (make-node
;                  :weights_list weights_list
;                  :input_list input_list)))

;; I now need to crate a set of functions to calculate the output of the neurone.
;; function a takes the weights and the input and sums w1*i1+..+wn*in
;; returning the value of a for the given node.

;(defun a (weights_list input_list)
;   (cond ((not (equal (length weights_list)(length input_list))) 'error)
;         ((null weights_list) 0)
;         (t (+ (a (cdr weights_list)(cdr input_list))
;              (* (car weights_list)(nth (car input_list) *structure*))))))

(defun a (node activity)
  (declare (type node node)
           (type (simple-array single-float (*)) activity))
  (let ((wa (node-weights-array node))
        (ia (node-input-array node))
        (sum 0.0s0))
    (declare (type (simple-array single-float (*)) wa)
             (type (simple-array fixnum (*)) ia)
             (type single-float sum))
    (when (not (= (length wa) (length ia)))
      (error "incompatible thingies"))
    (dotimes (i (length wa) sum)
      (declare (type fixnum i))
      (setf sum (+ sum
                   (* (aref wa i)
                      (aref activity (aref ia i))))))))

;;(a '(1 2 3) '(0 1 5))
;;>1

;; Now we need a sygmoid function to work out y given a as input

;(defun y (a)
;  (/ 1 (+ (exp (- a)) 1)))

(declaim (inline y))

(defun y (a)
  ;; I can't get allegro to compile this well (but I haven't tried
  ;; very hard). CMUCL does really well.
  (declare (type single-float a))
  (/ 1.0s0 (+ (exp (- a)) 1.0s0)))

;;(y 0)
;;> 0.5
;;(y 1)
;;> 0.73105...
;;(y -0.7)
;;> 0.331812227831834

;; Now we need a function that will take the current *node* structure and return y

;(defun Run_node ()
;  (y (a (node-weights_list *node*)(node-input_list *node*))))

(declaim (inline run-node))

(defun run-node (node activity)
  (declare (type node node)
           (type (simple-array single-float (*)) activity))
  (y (a node activity)))

;;(mk_node_struct '(2 3 6 -3) '(0 1 2 3))
;;>#S(NODE WEIGHTS_LIST (2 3 6 -3) INPUT_LIST (0 1 2 3))
;;(run_node)

;;> 0.880797077977882

;; Right, now that we have functions to work out the output of a given node, we now need to update
;; the *structure* with this new output value, and go through for each node

;(defun full_pass (node_struct_list)
;   (cond ((null node_struct_list) *structure*)
;         (t (dotimes (x (length node_struct_list) *structure*)
;               (setf *node* (nth x node_struct_list))
;             (setf (nth (+ x 1) *structure*)(run_node))))))

(defun full-pass (node-list activity)
  (declare (type (simple-array fixnum (*)) activity))
  (loop for i of-type fixnum upfrom 0
      for n of-type node in node-list
      do
        (setf (aref activity i) (run-node n activity))))

[snip]
    Tim> (defun y (a)
    Tim>   ;; I can't get allegro to compile this well (but I haven't tried
    Tim>   ;; very hard). CMUCL does really well.
    Tim>   (declare (type single-float a))
    Tim>   (/ 1.0s0 (+ (exp (- a)) 1.0s0)))

A common typo:  1.0s0 is a short-float, not a single-float.  This
doesn't matter to CMUCL since short-float and single-float are the
same, but it might matter to others.

In LISP, my understanding is that lists are held together with cons
cells:

   ---------------      ---------------
  | data* | next* | -> | data* | next* |
   ---------------      ---------------
      |  
   ---------------------------------
  | data | more data | and so forth |
   ---------------------------------

So once you get your hands on the list cell in C++ you have the data.
In LISP you have one more pointer deference to do.

Can you tell me your rationale for the LISP speedup?

    Tim> Are there any common platforms where short and single floats are
    Tim> different?  (I'm not trying to deny it was a mistake, I can never
    Tim> remember what the syntax for a short-float is though...)

Likewise.

CLISP does:

[1]> (float-digits 1s0)
17
[2]> (float-digits 1f0)
24

Don't know of any others.

In Macintosh Common Lisp, as of version 4.1, a short-float is an
IEEE single precision quantity (32 bits).  The rest of the float
types (single-float, double-float, long-float) are IEEE double
precision.

        So, the speedup for LISP is actually quite plausible.

        //Raymond.

Hoever, I don't think the issue of a pointer dereference is worth
thinking about. In my experience Lisp is much faster than C++ for list
management for several reasons.

Memory management: lisp systems are optimized to allocate small pieces
for lists, unlike "new" or "malloc" which are usually far slower than
CONS. Also, CONS will often be inlined, even further speeding things
up. (I have never seen new or malloc inlined in C or C++).
A good garbage collector should be able to free large numbers of list
elements much faster than calling free() on each one explicitly.

Because lisp is designed and optimized for list processing, operations
which traverse and alter list structures tend to be highly optimized
and inlined. Many standard lisp benchmarks stress these operations,
and compiler vendors have responded by using every possible trick to
make lists as fast as possible.

Of course in C++ you could write your own memory management functions,
and write your own macros for handling lists, and approach the
efficiency of lisp if you spend enough time on it. I think of this as
basically the "I can build Lisp in C++, therefore C++ can do anything
Lisp can do" argument.

Well, I could write a C++ compiler in Lisp...  :-)