(my-read-from-string "I am a string.")
=> "I"
   "am"
   "a"
   "string"
   "."

Or have it build a list. Which ever is more efficient:
(my-read-from-string "I am a string.")
=> ("I" "am" "a" "string" ".")

`read-from-string' uppercases everything, and returns `STRING.' as the last
word. I've been trying at this for awhile now and it seems that the only
function that can do what I want would have to understand the list of
alphabetic characters a-z and A-Z and use `subseq'. Is there any built in
function that almost does what I want, or any pointers about how to do this
most efficiently, and what to name it when it's done? The iterative `subseq'
version of this function would be enormous if I got it working right and make
a lot of garbage.

      (funcall #'values  seq-of-values )

   anyway.  Since the number of values varies here.
   So the latter seems the better way to go.

    I'm afraid you'll need to write you own lexer.
    (i.e.  walk down the string testing to see word breaks (non alpha-char-p )

          012345678901234
         "I am a string."

USER(1): (subseq "I am a string." 0 1 )
"I"
USER(2): (subseq "I am a string." 2 4 )
"am"
USER(3): (subseq "I am a string." 5 6 )
"a"
USER(4): (subseq "I am a string." 7 13 )
"string"
USER(5): (subseq "I am a string." 13 14 )
"."

     [ This is probably an O(n^2) process since subseq must skip
        the ever lenghtening prefix each time.... if subseq is
        naive about the sequence in question.  Probably not worth
        worrying about. ]

     If you're not so concerned with garbage then  string-trim
     ( string-trim-left) could help tokenize the given string.

--

In MCL 4.2, the boundary seems to be at a "word" size of 20 characters. The
actual allocation has to do with granularity of memory requests and the
relative sizes of simple vs non-simple array headers.

So if I was going to use MCL 4.2 to chunk English text and was concerned
about total memory allocation, the subseq version would save me about 8
bytes per word.

OTOH, if I was more concerned about stressing the GC, I'd probably
preallocate a bunch of displaced arrays, and change the code slightly to use
them as resources (rather than creating them for each use and then
discarding them).

;; Tests of subseq version

? (let ((string "T "))
    (time (read-short-words-from-string string)))
(READ-SHORT-WORDS-FROM-STRING STRING) took 0 milliseconds (0.000 seconds) to
run.
 16 bytes of memory allocated.
("T")

? (let ((string "ThisThat "))
    (time (read-short-words-from-string string)))
(READ-SHORT-WORDS-FROM-STRING STRING) took 0 milliseconds (0.000 seconds) to
run.
 24 bytes of memory allocated.
("ThisThat")

? (let ((string "ThisThatThisThatThis "))
    (time (read-short-words-from-string string)))
(READ-SHORT-WORDS-FROM-STRING STRING) took 0 milliseconds (0.000 seconds) to
run.
 32 bytes of memory allocated.
("ThisThatThisThatThis")

? (let ((string "ThisThatThisThatThisX "))
    (time (read-short-words-from-string string)))
(READ-SHORT-WORDS-FROM-STRING STRING) took 0 milliseconds (0.000 seconds) to
run.
 40 bytes of memory allocated.
("ThisThatThisThatThisX")

;; Tests of displaced arrays version

? (let ((string "ThisThatThisThatThisX "))
    (time (read-long-words-from-string string)))
(READ-LONG-WORDS-FROM-STRING STRING) took 0 milliseconds (0.000 seconds) to
run.
 32 bytes of memory allocated.
("ThisThatThisThatThisX")

? (let ((string "ThisThatThisThatThisThisThatThisThatThisThisThatThisThat"))
    (time (read-long-words-from-string string)))
(READ-LONG-WORDS-FROM-STRING STRING) took 0 milliseconds (0.000 seconds) to
run.
 32 bytes of memory allocated.
("ThisThatThisThatThisThisThatThisThatThisThisThatThisThat")

;; Note: These only recognize words composed of alpha characters, and
;; require that the last word be followed by a non-alpha character.
;; Marking the "word" boundaries properly is left as an exercise. <g>

(defun read-long-words-from-string (string)
  (let ((result ())
        (in-word-p nil)
        (start-word nil))
    (dotimes (i (length string))
      (setq in-word-p (alpha-char-p (char string i)))
      (if start-word
        (unless in-word-p
          (push (make-array (- i start-word)
                            :displaced-to string
                            :displaced-index-offset start-word)
                result)
          (setq start-word nil))
        (when in-word-p
          (setq start-word i))))
    (nreverse result)))

(defun read-short-words-from-string (string)
  (let ((result ())
        (in-word-p nil)
        (start-word nil))
    (dotimes (i (length string))
      (setq in-word-p (alpha-char-p (char string i)))
      (if start-word
        (unless in-word-p
          (push (subseq string start-word i)
                result)
          (setq start-word nil))
        (when in-word-p
          (setq start-word i))))
    (nreverse result)))

--
David B. Lamkins <http://www.teleport.com/~dlamkins/>

[snip]

Therefore, subseq allocates additional memory (which may eventually turn
into garbage).

[snip]

[snip]

--
--
David B. Lamkins <http://www.teleport.com/~dlamkins/>

        (apply #'values seq-of-values)

which is equivalent to:

        (values-list seq-of-values)

--

    which does create some intermediate garabge. ]

    If you later wished to flip these strings into symbols you'd need
    to (or should ) make a copy of the strings anyway.

    Additionally if the "source string" is some reusable buffer then
    you can just refill the next string. As opposed to creating a new
    string for "line".  [ e.g., using read-sequence to fill/refill the buffer
    in constrast to repeatively calling read-line. ]

    Most everything eventually turns into garbage.  Even the aliases.

    Also, if your "source" string consists of lots of "white space" then
    if you let it go and just keep the tokens you have a net space
    reduction.

--

(defun substr (string from &optional (to (length string)))
  (make-array (- to from) :element-type 'character
              :displaced-to string
              :displaced-index-offset from))

However, with ACL 5, the initial overhead of this function compared
to subseq seems to be 32 bytes, which means, I guess, that you'll
need to have substrings longer than than 32 characters to make any
sensible use of it...

--

(defun split (string &optional max (ws '(#\Space #\Tab)))
  "Split `string' along whitespace as defined by the sequence `ws'.
The whitespace is elided from the result.  The whole string will be
split, unless `max' is a non-negative integer, in which case the
string will be split into `max' tokens at most, the last one
containing the whole rest of the given `string', if any."
  (flet ((is-ws (char) (find char ws)))
    (loop for start = (position-if-not #'is-ws string)
          then (position-if-not #'is-ws string :start index)
          for index = (and start
                           (if (and max (= (1+ word-count) max))
                               nil
                               (position-if #'is-ws string :start start)))
          while start
          collect (subseq string start index)
          count 1 into word-count
          while index)))

Note:

a) This is in no way optimized.  Depending on your usage patterns and
   your implementation this can be tuned quite a lot.  Modify it first
   to meet your needs, use it, if it's too slow, profile it (this can
   have very interesting ramifications!), and then optimize it, if
   need be.
b) The documentation string is slightly dense...

Regs, Pierre.

PS: I still think that these kind of string-processing functions
should be collected somewhere in a nice library, so that new users
will not find Perl more suitable for simple string-bashing things...
As time permits (which it seldomly does), I'll try to put something
usable together, and release it to the PD.  Anyone who has something
nice to contribute, feel free to contact me... ;)

While looking through the HyperSpec just recently I found this
TOKENIZE-SENTENCE function on the LOOP-FINISH page:

;; While LOOP-FINISH can be used can be used in a variety of
;; situations it is really most needed in a situation where a need
;; to exit is detected at other than the loop's `top level'
;; (where UNTIL or WHEN often work just as well), or where some
;; computation must occur between the point where a need to exit is
;; detected and the point where the exit actually occurs.  For example:
 (defun tokenize-sentence (string)
   (macrolet ((add-word (wvar svar)
                `(when ,wvar
                   (push (coerce (nreverse ,wvar) 'string) ,svar)
                   (setq ,wvar nil))))
     (loop with word = '() and sentence = '() and endpos = nil
           for i below (length string)
           do (let ((char (aref string i)))
                (case char
                  (#\Space (add-word word sentence))
                  (#\. (setq endpos (1+ i)) (loop-finish))
                  (otherwise (push char word))))
           finally (add-word word sentence)
                   (return (values (nreverse sentence) endpos)))))
=>  TOKENIZE-SENTENCE

 (tokenize-sentence "this is a sentence. this is another sentence.")
=>  ("this" "is" "a" "sentence"), 19

 (tokenize-sentence "this is a sentence")
=>  ("this" "is" "a" "sentence"), NIL

Does anyone know of a strings package that has already been put
together by a vendor, and where API-related documentation for it might
be accessed?

If we don't have to reinvent the wheel here and have two incompatible
interfaces, that would be wonderful.

This is a very serious issue.  These functions should be
implementation-depenedent, and they should be completely PD, IMO.
Anyway, I began working on this years ago in elisp (i.e. trying to write
many Perl functions in Lisp), but I'll have to find it, and when/if I
do, I'll also make it public.  My `split' was almost a facsimile of
Pierre's, and I think that it should not be too hard to do this
(although to do it _right_ (e.g. extensible, versatile, etc) is another
story altogether).

For efficiency, use C/C++.

Parentheses have been known to go awry during cut and paste...

Hello "new troll on the block"!

(defun string-token (string token)
  "DESCRIPTION:   Tokenize a string similar to the C string library
                  function strtkn. Locates the position p of any
                  member of list token within the input string and returns
                  the substring consisting of the characters from
                  position 0 to p-1 and a substring from p+1 to the end of the
                  input string. If no member of token is not found, the
                  string and nil are returned.

                    Examples:
                      (string-token \"frog-went-a-courtin'\" #\-)
                         =>\"frog\"
                           \"went-a-courtin'\"

                      (string-token \"frog-went-a-courtin'\" #\z)
                         =>\"frog-went-a-courtin'\"
                            NIL

                      (string-token \"frog-went-a-courtin'\" (list #\g #\a))
                         =>\"fro\"
                           \"-went-a-courtin'\"

                      (string-token \"frog-went-a-courtin'\" #\f)
                         =>\"\"
                           \"rog-went-a-courtin'\"

     PARAMETERS:    string         -string-
                    token          -character or list of characters-"

  (if (and string token)
      (do* ((delim-list (if (and token (listp token)) token (list token)))
            (char-list (coerce string 'list))
            (limit (list-length char-list))
            (char-count 0 (+ 1 char-count))
            (char (car char-list) (nth char-count char-list))
            )
           ((or (member char delim-list)
                (= char-count limit))
            (return
              (values-list
               (if (= char-count limit)
                   (list string nil)
                   (list (coerce (butlast char-list (- limit char-count))
                                 'string)
                         (coerce (nthcdr (+ char-count 1) char-list) 'string))
                   )))))))

If you do not understand why it is better, too bad!

Note also that, for your instruction, your code could be rewritten as follows
(there is one minor difference: where is it?)

(defun string-token (string &optional (separators '(#\Space)))
  (let ((p (position-if #'(lambda (c) (member c separators :test #'char=))
                        string)))
   (if (null p)
       str
       (values (subseq str 0 p)
               (subseq str (1+ p))))))

which is equivalent, faster and less memory hungry.  No wonder your
Lisp code is so slow.

Moreover, what you usually really want is

(defun string-split (str &optional (separator #\Space))
  "Splits the string STR at each SEPARATOR character occurrence.
The resulting substrings are collected into a list which is returned.
A SEPARATOR at the beginning or at the end of the string STR results
in an empty string in the first or last position of the list
returned."
  (declare (type string str)
           (type character separator))
  (loop for start  = 0 then (1+ end)
        for end    = (position separator str :start 0)
                     then (position separator str :start start)
        for substr = (subseq str start end)
                     then (subseq str start end)
        collect substr into result
        when (null end) do (return result)
        ))

However, in order to achieve the same effect in C/C++ you'd have to
manually allocate (i.e. with 'malloc') some arrays. Which in *your*
case would probably result in memory leaks around your application
too.

Finally, what is usually very useful is something more complex that
implements a tokenizer in the spirit of Java StringTokenizer class. (I
have it. Do you want it?)

Cheers

If the previous seems a little complicated for the beginners, there are
other ways of doing this, too. For example:

(defun string-token-B (str token)
  (let ((pos (position token str)))
    (if (null pos)
        (values str nil)
      (values (subseq str 0 pos)
              (subseq str (1+ pos))))))

-- simplified version: only accepts single tokens, no input sanity
checking. Seems to be slightly faster, too.

  well, the interesting question is how _did_ you survive?  if you arranged
  deck chairs the way you write code, it's no wonder the whole ship sank.

Steve Long <stevel...@isomedia.com> is apparently trolling in the lisp
groups.  He says:

 Here's a way to break up strings.

And translates his C code into lisp (badly).  As others have pointed out,
the whole thing could be done better and much more simply with POSITION and
SUBSEQ.  I want to add that I write parsers in lisp all the time (usually to
process human-friendly commands to complex systems).  Here's the lexer I
use, which does a good bit more than Steve's tokenizer, and is quite
efficient.

(defun lex-string (string &key (whitespace '(#\space #\newline)))
  "Separates a string at whitespace and returns a list of strings"
  (flet ((whitespace? (char) (member char whitespace :test #'char=)))
    (let ((tokens nil))
      (do* ((token-start
             (position-if-not #'whitespace? string)
             (when token-end
               (position-if-not #'whitespace? string :start (1+ token-end))))
            (token-end
             (when token-start
               (position-if #'whitespace? string :start token-start))
             (when token-start
               (position-if #'whitespace? string :start token-start))))
           ((null token-start) (nreverse tokens))
        (push (subseq string token-start token-end) tokens)))))

There's always a better way of doing things, especially in a LISP
newsgroup.

  no, they haven't, but it's actually comforting to learn that you think so.