reading a file into a string
Dr. Edmund Weitz
would be the best way to do it.
With CMUCL I would to do something like
(defun file-size (file-name)
(let* ((result (multiple-value-list (unix:unix-stat file-name)))
(ok (car result)))
(if ok
(nth 8 result)
(error "Couldn't stat file ~S" file-name))))
(defun string-from-file (file-name &key (max-size 4096))
(let ((size (file-size file-name)))
(when (> size max-size)
(error "File ~S is too large" file-name))
(let ((string (make-string size)))
(with-open-file (stream file-name)
(read-sequence string stream))
string)))
My questions are:
1. Is this solution efficient and OK or are there any significant
reasons to use a different approach?
2. Is there an ANSI-compliant solution that is comparable in
performance? Maybe something along the lines of Bulent
Murtezaoglu's solution that can be found at
<http://www.bagley.org/~doug/shootout/bench/reversefile/reversefile.cmucl>?
Thanks,
Edi.
Nils Goesche
I once used the following code:
(defun get-input (str)
(let* ((size #x100000)
(ret (make-string size))
(write-pos (read-sequence ret str)))
(if (< write-pos size)
(subseq ret 0 write-pos)
(concatenate 'string ret (get-input str)))))
It seemed to be pretty fast to me; also, it is ANSI-compliant and
resistant to changes in the size of the file (without locking).
How about it?
Regards,
--
Nils Goesche
Ask not for whom the <CONTROL-G> tolls.
PGP key ID 0xC66D6E6F
Bulent Murtezaoglu
[...]
EW> My questions are:
EW> 1. Is this solution efficient and OK or are there any
EW> significant reasons to use a different approach?
I think read-sequence, with the right declarations, can be optimized into
a read syscall. That's as good as it's going to get. So I'd be surpised
if your approach could be bettered.
EW> 2. Is there an ANSI-compliant solution that is comparable in
EW> performance? Maybe something along the lines of Bulent
EW> Murtezaoglu's solution that can be found at
EW> <http://www.bagley.org/~doug/shootout/bench/reversefile/reversefile.cmucl>?
I don't think I wrote all of that -- Doug Bagley must have dropped some
attributions along the way. At any rate it does not look like something
I'd write from scratch (too loopy to come naturally to me). I remember
sending him sth. based on a variant of my with-fast-string-reader
macro at some point though
The problem in the shootout entailed reading from a pipe, so we could not
set the file (and thus the string) size in advance. Since you state you are
reading from a real file, things are easy to do within ANSI CL (see
file-length in the hyperspec as your stat call replacement).
cheers,
BM
Bulent Murtezaoglu
[...]
DB> Sorry, I think it has bits of me and Paul Foley in it :-)
Sounds right. Sorry Paul! I must have monkeyed with it enough for you think
I wrote it all.
>> ... (see file-length in the hyperspec as
>> your stat call replacement).
DB> But that has a race condition if the file changes as Nils
DB> points out.
That is correct, of course. The stat call Dr. Weitz had suffers from
the same shortcoming. I wouldn't have thought of that had I not seen
NG's posting. I think NG's code might be too consy (especially if the
heap isn't in the right state) because of the concatenate. I also
noticed this in the C code in your shootout that came out on top partially
because realloc was being called with a mostly virgin heap. This, I am
basing on no experimentation whatsoever of course!
Might I also point out we are BSing about semi-pointless CL hacks on a Friday
night on cll. We should remember this thread the next time the is Lisp Dead
thread comes up.
cheers,
BM
Bulent Murtezaoglu
BM> ... I think NG's code might be
BM> too consy (especially if the heap isn't in the right state)
BM> because of the concatenate. ...
Just to speak to myself some. It is guaranteed to cons. I just looked up
concatenate. It cannot touch its arguments. I was confusing it with some
figment of my imagination (ie I dunno what it's called if if it exists).
cheers,
BM
Nils Goesche
Well, it's indeed Friday night and I am, despite my being a German,
working on my second bottle of Chablis right now, but IIRC, I timed
the code at the time I wrote it. It was surprisingly fast, *despite*
the apparent consing. That might have been because of the mighty
compiler writers, but who knows. Time it for what you need it
and you'll see what's best for you.
Darn, that second bottle is empty, too!
Erik Naggum
| I want to read a whole (text) file into a string. And I wonder what
| would be the best way to do it.
The function is named file-contents in Allegro CL and is based on an
earlier version of mine, which goes like this:
(defun file-contents (pathname &rest open-arguments)
"Map the current contents of the file named by PATHNAME into a vector,
which is returned. OPEN-ARGUMENTS is a list of arguments passed to OPEN;
it should not include a :DIRECTION argument. An :ELEMENT-TYPE argument
also specifies the type of the returned vector. Returns NIL if the file
does not exist and :IF-DOES-NOT-EXIST is NIL."
(with-open-stream (stream (apply #'open pathname :direction :input open-arguments))
(when stream
(let ((buffer (make-array (file-length stream) :element-type (stream-element-type stream))))
(if (= (read-sequence buffer stream) (length buffer))
buffer
(error 'file-error
:format-control "Incomplete READ-SEQUENCE from ~S."
:format-arguments (list (pathname stream))
:pathname (pathname stream)))))))
(defun (setf file-contents) (vector pathname &rest open-arguments)
"Make the contents of the file named by PATHNAME become that of VECTOR,
which is returned. OPEN-ARGUMENTS is a list of arguments passed to OPEN;
it should not include either a :DIRECTION or an :ELEMENT-TYPE argument.
Does nothing and returns NIL if the file exists and :IF-EXISTS is NIL.
:IF-EXISTS defaults to :SUPERSEDE."
(with-open-stream (stream (apply #'open pathname
:direction :output
:element-type (array-element-type vector)
(append open-arguments '(:if-exists :supersede))))
(when stream
(write-sequence vector stream))))
Allegro CL has very strong support for various external-formats, so if
you specify (or default to) characater as the element type, this may do
some work between the read system call and read-sequence, so although it
is very fast and efficient, it is hard to beat a byte-for-byte copy --
check which external-formats are available and apply If you want
byte-for-byte copies, use (unsigned-byte 8) as the element-type. The
portable way to convert between (unsigned-byte 8) and character is to use
(map #'string #'code-char <byte-vector>), but you can usually find a way
to hack the type system so such a byte vector turns into a string, but
this is not portable, of course. Please remember that element-type and
external-format are available keyword arguments to open.
| 2. Is there an ANSI-compliant solution that is comparable in
| performance?
The above should be. Notice that file-length is a standard function, but
it applies to a stream, not to pathnames.
/// 2002-02-02
--
In a fight against something, the fight has value, victory has none.
In a fight for something, the fight is a loss, victory merely relief.
Thomas A. Russ
> 2. Is there an ANSI-compliant solution that is comparable in
> performance? Maybe something along the lines of Bulent
> Murtezaoglu's solution that can be found at
> <http://www.bagley.org/~doug/shootout/bench/reversefile/reversefile.cmucl>?
The only tricky part is that the ANSI file-length operates on an
open file stream rather than on the file-name, so we just need to
reorder a few elements in your reader function:
(defun string-from-file (file-name &key (max-size 4096))
(with-open-file (stream file-name)
(let ((size (file-length stream)))
(when (> size max-size)
(error "File ~S is too large" file-name))
(let ((string (make-string size)))
(read-sequence string stream))
string)))
--
Thomas A. Russ, USC/Information Sciences Institute t...@isi.edu
Bulent Murtezaoglu
[...]
NG> Well, it's indeed Friday night and I am, despite my being a
NG> German, working on my second bottle of Chablis right now,
Well, just as one might expect from a Turk, I'm having coffee and
smoking here...
NG> but IIRC, I timed the code at the time I wrote it. It was
NG> surprisingly fast, *despite* the apparent consing. That might
NG> have been because of the mighty compiler writers, but who
NG> knows. Time it for what you need it and you'll see what's
NG> best for you.
I think this is the more important point. Outside of microbenchmarking
and for small enough files I think it is fine. But since we've gone this
far, I actually went ahead and benchmarked things including C.
On a P-III 933, with 512Meg, Linux 2.4.17
CMUCL (Debian, PvE version 3.0.8 18c+ 31 December 2001 build 3030)
File's in buffer cache.
ftest1 is based on your code, ftest2 is file-length/read-sequence
combination, C is the attached C code. All times are wall clock time as
reported by time. There wasn't much variance among multiple runs.
Of course, how often do you read files that have actually been touched
recently enough to be in buffer cache? That is, outside of stupid
microbenchmarking.
File size: 1 Mbyte
ftest1 : 0.01s, 2048608 bytes consed
ftest2 : 0.01s, 1009160 bytes consed
C : 0.008s
File size: 10 Mbyte
ftest1 : 0.58s, 63300680 bytes consed
ftest2 : 0.07s, 10007760 bytes consed
C : 0.065s
File size: 50 Mbyte
ftest1 : (*)
ftest2 : 0.33s, 50000016 bytes consed (**)
C : 0.310s
(*) I got
Type-error in KERNEL::OBJECT-NOT-TYPE-ERROR-HANDLER:
139953152 is not of type (UNSIGNED-BYTE 27)
from KERNEL:BIT-BASH-COPY 5
(**) This can be make to match the C time probably by tweaking the GC not to
kick in when the huge string is allocated.
Here's the code:
(defconstant fname #p"/tmp/rtest")
(declaim (optimize (speed 3) (safety 0) (debug 0)))
(defun get-input (str) ;; off NG's posting
(let* ((size #x100000)
(ret (make-string size))
(write-pos (read-sequence ret str)))
(if (< write-pos size)
(subseq ret 0 write-pos)
(concatenate 'string ret (get-input str)))))
(defun ftest1 ()
(with-open-file (str fname)
(let ((buffer (get-input str)))
(format t "~d bytes read" (length buffer)))))
(defun ftest2 ()
(with-open-file (str fname)
(let ((buffer (make-string (file-length str))))
(format t "~d bytes read" (read-sequence buffer str)))))
and here's the C code:
#include <stdio.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <unistd.h>
const char fn[]="/tmp/rtest";
int main()
{
char * buff;
struct stat statbuf;
unsigned long int fsize, readsize;
FILE *infile;
if (stat(fn, &statbuf) == -1) { /* is stat in ANSI C lib yet? */
printf("Cat't stat \n");
exit(1);
}
fsize = statbuf.st_size; /* assume file */
buff = malloc(fsize, 1);
if (buff == NULL) {
printf("Couldn't malloc \n");
exit(1);
}
infile = fopen(fn, "r");
if (fsize == (readsize = fread(buff, 1, fsize, infile))) {
printf("Read %d bytes \n", readsize);
} else {
printf("Read only %d bytes \n", readsize);
}
}
Nils Goesche
>>>>>> "NG" == Nils Goesche <car...@t-online.de> writes:
>
> [...]
>
> NG> Well, it's indeed Friday night and I am, despite my being a
> NG> German, working on my second bottle of Chablis right now,
>
> Well, just as one might expect from a Turk, I'm having coffee and
> smoking here...
That's what I do the rest of the week. The Turks are probably the
only ones left in Germany who do any real work. Besides me, that
is.
> NG> but IIRC, I timed the code at the time I wrote it. It was
> NG> surprisingly fast, *despite* the apparent consing. That might
> NG> have been because of the mighty compiler writers, but who
> NG> knows. Time it for what you need it and you'll see what's
> NG> best for you.
>
> I think this is the more important point. Outside of microbenchmarking
> and for small enough files I think it is fine. But since we've gone this
> far, I actually went ahead and benchmarked things including C.
I just can't resist whenever anybody give results of whatever little
microbenchmark they did. So I repeated your experiments. And indeed,
I got similar results. Ok, let's have #'get-input move into the
orcus forever :-)
Dr. Edmund Weitz
> * Dr. Edmund Weitz
> | I want to read a whole (text) file into a string. And I wonder
> | what would be the best way to do it.
>
> The function is named file-contents in Allegro CL and is based on
> an earlier version of mine, which goes like this:
>
> [...]
Thank you very much, and also thanks to all the other people who sent
insightful and interesting answers to c.l.l or privately by email -
I've definitely learned a couple of things.
As for FILE-LENGTH and my resorting to UNIX:UNIX-STAT: I knew
something like that had to be in the standard as CL seems to have a
function for every task I can imagine and then some. I think the
problem was that I was in a situation similar to that of Nils: My wife
and I had just finished our second bottle of Gavi and so I just
scanned the files dictionary in the HyperSpec while it didn't occur to
me that I might also scan the chapter about streams or just try my
luck with APROPOS. Don't know how to fix this problem - omitting the
Italian white wine doesn't seem to be an option... :)
Thanks again,
Edi.
Kent M Pitman
> I want to read a whole (text) file into a string.
Why?
While it's useful to know how to do various things in a language,
questions of this sort are usually supposed to trigger questions
like "what are you REALLY trying to do?"
Reading a whole file (which may be of arbitrary size) into memory
(which is usually of finite size) usually suggests you're going to
randomly access it. Suppose that it takes 1 unit of time to move each
byte from file to memory. Suppose it takes another 1 unit to read
from the array. Suppose that the GC time for the array is another 1
unit per cell. Suppose further that while the file is in-memory, GC
frequency increases because the in-memory data is more crowded. And
suppose file I/O takes 5 units per character. Then basically, it
looks like you've nothing for trying. Or at least you'd better be
planning to touch most of the file or you'd better need substantial
"real time" response to pay for all the setup work.
It wouldn't surprise me if opening the file in direct access mode,
which you can do in some implementations, even though it's not
standard, wouldn't get you better performance. And in other cases,
just doing some manual buffering of small sections of the file at once
would seem better.
But I think randomly offering people "how to" information in response
to questions like this is a Bad Plan. While CL supports the ability
to do what you want, I think it intends you do to
FILE-POSITION + READ-BYTE.
You also used the "performance" without reference to a program,
algorithm, or data set. That's a no-no. Performance is a vector,
not scalar, quantity. Performance of setup is different than reference
performance, and reference performance is a function of data,
implementation, etc.
Tim Bradshaw
> While it's useful to know how to do various things in a language,
> questions of this sort are usually supposed to trigger questions
> like "what are you REALLY trying to do?"
One reason, it seems to me, for doing this kind of thing is caching
files for things like web servers. Modern (Unix, don't know about
Windows, but I assume there too) filesystems can get *very* good
performance if you give the machine sensible memory, but if you want
to spit out the contents of a file you still have to do things like
translate the filename to get the file, make a system call to read the
file and then another to write it. Of course these things often
involve much less work than you might thing because the file is
buffered in memory and the system calls just mmap it, but even then
it's hard to beat `look up the name in a table and spit out the vector
you've just got', which might easily be one system call. This can be
especially true if there are nontrivial encodings in the file (if it's
UTF-8 or something), when you can cache the translated contents as
bytes which you just spit down the wire.
--tim
Thaddeus L Olczyk
wrote:
>d...@agharta.de (Dr. Edmund Weitz) writes:
>
>> I want to read a whole (text) file into a string.
>
>Why?
I can give one example that happened to me recently.
I had to extract certain strings from a file.
These strings were too complex to write a simple search algorithm.
They were too simple for a parser.
They were just right for a regular expresion match.
One problem was that the expressions that matched would extend across
lines.
Most of the files would be relatively small, maybe one file
conceivable be 10M but 99.9% would run less then 50K and
95% would be less then 5K. So it was simpler to just read the whole
file in.
Dr. Edmund Weitz
Thanks for your reply. I'll talk about why I (think I) need or want
this feature below. But first allow me some general remarks:
The fact that someone is new to Common Lisp and doesn't know all of
its features doesn't necessarily imply that this person is completely
new to computers, has never heard of algorithms before or is otherwise
slightly behind the times. I, for example, have written my first
Assembler programs when I was 16 years old, and I'm 36 now. I've been
working as a professional programmer for a couple of years and
this is not the first time that I was confronted with the question
whether it would be better to read a file at once or rather to read it
line by line, do manual buffering or use fseek-like features.
Also, I didn't actually ask for "how to" information but presented a
working implementation that I intended to use and asked the CL experts
if I had missed something important.
Holding back information because you think that it might be "too
dangerous" for the recipient or that he might not me mature or
knowledgeable enough to use it correctly seems like a bad concept to
me. This would be like leaving out a couple of chapters from the
HyperSpec and only make them available to people who can prove that
they have successfully used CL for at least five years... :)
Having said that, here's some background for my problem:
I spend most of my professional live doing the background programming
for dynamic, database-backed websites, i.e. I fiddle around with
Apache, Oracle or PostgreSQL and have to use inferior programming
languages like C, Java, Perl, or (yuck!) PHP. There are lots of
clients where I can't choose the programming environment but
_sometimes_ I work on smaller projects where I actually have the last
word about which tools to use.
As I'm beginning to like CL more and more each day, I started to
contemplate if it wouldn't be a good idea to use Lisp for my web
projects and not only for the fun of it. So I began to evaluate a
couple of tools that I found on the web like UncommonSQL and
(Portable) AllegroServe and was very pleased with what I saw. There
was only one area where I wasn't convinced with the tools available
and that was HTML generation.[1]
There do exist a couple of very sophisticated libraries like onShore's
IMHO or AllegroServe's HTML generation facility, to name only two, but
(at least for my situation) they seem to have the wrong approach: They
put the task of generating the complete HTML page into the
program. Almost every project that I was involved in had one or more
graphic designers who were responsible for spewing out the HTML while
developers like me used these templates and added the application
logic, so putting the HTML into the program, whether Lisp or not,
would not be an option.
The other extreme is to embed the programming language into the HTML
like with ASP, JSP, PHP, and so on. From my experience this leads to
code that is very hard to read and maintain and apart from that is
also simply unesthetic.[2]
The approach that worked best for me is to separate HTML and
application as far as possible and leave only a minimum of 'code' in
the HTML template, so when I could choose which approach to use I
usually opted for Sam Tregar's HTML::Template
<http://sourceforge.net/projects/html-template/>. (Forget for a
moment, if you can, that it's based on Perl... :)
If you're still with me, here's what I want to do: I want to implement
something similar in Lisp, i.e. a program that scans a HTML template
file containing a couple of simple tags like
<!-- TMPL_VAR NAME=foo -->
and replaces these tags with output according to the current state of
the application.
While the original HTML::Template module (AFAIK) optionally caches the
template file in RAM, it parses it every time it is invoked in order
to do the replacement. I think CL can do it better. To simplify a bit:
Given a template file that contains
bar <!-- TMPL_VAR NAME=foo --> baz
I want to automatically generate a function like
(lambda (foo)
(princ "bar ")
(princ foo)
(princ " baz"))
and optionally compile it, i.e. a production system will have one
function per dynamic page, and each template has to be parsed only
once. It should be clear from this that I have to have the whole file
in RAM anyway.
As for the feasibility of this approach: I have checked a large
application of one of my customers. It consists of about 3,000 HTML
files most of which are from 1kB to 3kB in size, many of them
containing lots of PHP code.[3] Also, about 30% of these pages are
completely static and can be served by a vanilla Apache or the new
kernel-based Linux httpd that I forgot the name of.
But let's suppose all 3,000 of them were 4kB in size and we would have
_all_ of them in memory. Let's also suppose that we have to multiply
this by a factor of 2 for the functions that are generated. This would
result in 24MB which is not that much given the fact that this
application runs on a couple of Solaris boxes with 1GB to 4GB RAM
each...[4]
Also, I've had many situations where we didn't have 2,000 but rather
less than 100 dynamic pages, so the memory requirements would be
negligible in this case - even my laptop has 256MB RAM...
I hope this makes clear why I think that my approach of reading the
whole file into memory might be OK in this case.
Apart from that, I'd be happy to further discuss my thoughts about
dynamic HTML generation in general and with CL in particular with
anyone who's interested although this might be slightly off-topic in
this newsgroup.
> You also used the "performance" without reference to a program,
> algorithm, or data set. That's a no-no. Performance is a vector,
> not scalar, quantity. Performance of setup is different than
> reference performance, and reference performance is a function of
> data, implementation, etc.
Pardon me? I presented a function that did exactly one thing - read a
file into a string. It was also easy to see that I considered 4kB
files to be very large. So the definition of performance in this
particular context seems rather clear to me and a couple of other
posters were obviously able to understand me.
Thanks for your time,
Edi.
[1] Note: Most of what I'm saying here is not CL-specific. IMHO it's a
general problem of how to integrate HTML with application logic.
[2] I don't buy the current XSLT hype either.
[3] You might remember that my original posting considered a file
above 4k as too large by default.
[4] Here I'm thinking about a CL web server like AllegroServe of
course, where the functions would be shared between the child
servers. If we had a setup with something like Apache and mod_perl
we would have to multiply the 24MB with the number of processes
which would be a disaster...
vc_sabe
> Kent M Pitman <pit...@world.std.com> writes:
>
> > e...@agharta.de (Dr. Edmund Weitz) writes:
> >
> > > I want to read a whole (text) file into a string.
> >
> > Why?
> >
)ssssnip)
> >
> > But I think randomly offering people "how to" information in response
> > to questions like this is a Bad Plan. While CL supports the ability
> > to do what you want, I think it intends you do to
> > FILE-POSITION + READ-BYTE.
>
> Thanks for your reply. I'll talk about why I (think I) need or want
> this feature below. But first allow me some general remarks:
>
)ssnip)
>
> Holding back information because you think that it might be "too
> dangerous" for the recipient or that he might not me mature or
> knowledgeable enough to use it correctly seems like a bad concept to
> me. This would be like leaving out a couple of chapters from the
> HyperSpec and only make them available to people who can prove that
> they have successfully used CL for at least five years... :)
>
Hi-
I think his point was not that you should hold back information from people, but
rather that in LISP a whole bunch of questions arise even if you do simple things. That is my
impression, and this probably has to do with the "worse is better debate". By that I mean
that some of these questions would not even be asked in some other programming language, or
at least the solution would be very limited.
Actually, I not so sure about what I'm talking about :-) Fact is, I'm a newbie. :-)
But is just a certain * feeling * one gets. Like when you read a book, a sentence. You
read code! Same thing...
Regs
synth...@uol.com.br
Wade Humeniuk
"Dr. Edmund Weitz" <e...@agharta.de> wrote in message
news:m3lmebi...@bird.agharta.de...
> Kent M Pitman <pit...@world.std.com> writes:
>
> If you're still with me, here's what I want to do: I want to implement
> something similar in Lisp, i.e. a program that scans a HTML template
> file containing a couple of simple tags like
>
> <!-- TMPL_VAR NAME=foo -->
>
> and replaces these tags with output according to the current state of
> the application.
>
I am pretty sure CL-HTTP has this functionality. The server expands special
tags into Lisp code. Not sure if it compiles it or not. The web site does
not
seem to working correctly at the moment.
What you are asking is not that difficult. Just suck everything which is
not the Lisp tag into a string until you hit the tag, then the code is
inserted. By why just insert the state of a var, why not insert Lisp code?
Example
<html>
<head><title>Test Page</title>
<body>
Here is some code
<lisp>(princ (* 100 300))</lisp>
</body>
</html>
read and translate it into an anonymous function
(lambda (&optional (stream *standard-output*))
(let ((*standard-output* stream)
(write-string "<html><head><title>Test Page</title><body>Here is some
code")
(princ (* 100 300))
(write-string "</body></html>"))
compile it, store it somewhere so your web server can retrieve it.
The main limitation with this solution, for example, is having the Lisp code
insert a variable number of rows into a table. The code needs facilities to
generate tags.
e.g.
<html>
<head><title>Test Page</title>
<body>
Here is a table
<table>
<lisp>(loop for i from 0 upto *number-of-members*
do
(write-string "<tr><td>")
(princ "member")
(princ i))
</lisp>
</table>
</body>
</html>
Then it gets ugly.
I am sure there are other issues.
Wade
Takehiko Abe
; (defun file-contents (pathname &rest open-arguments)
; [...]
; (with-open-stream (stream (apply #'open pathname
; :direction :input open-arguments))
; (when stream
; (let ((buffer (make-array (file-length stream)
; :element-type (stream-element-type stream))))
; (if (= (read-sequence buffer stream) (length buffer))
; buffer
; (error 'file-error
; :format-control "Incomplete READ-SEQUENCE from ~S."
; :format-arguments (list (pathname stream))
; :pathname (pathname stream)))))))
I think the use of format-control and format-arguments initargs
is not portable. Hyperspec doesn't list simple-condition in
class-precedence-list for file-error. CLISP (v.2.27) errored
with them.
regards,
abe
--
<keke at mac com>
Bulent Murtezaoglu
EW> application as far as possible and leave only a minimum of
EW> 'code' in the HTML template, so when I could choose which
EW> approach to use I usually opted for Sam Tregar's
EW> HTML::Template [...]
Yes this fits the situation you are describing. I agree that usually
one gets called in when the 'html authors' or designers or whatever have
already either been hired or have done work and got paid paid for it so
"let's generate all this from a program" doesn't fly. Thsi is not
necessarily a technical issue, but it is an important issue.
EW> If you're still with me, here's what I want to do: I want to
EW> implement something similar in Lisp, i.e. a program that scans
EW> a HTML template file containing a couple of simple tags like
EW> <!-- TMPL_VAR NAME=foo -->
EW> and replaces these tags with output according to the current
EW> state of the application.
Yes, but a Wade Humeniuk. pointed out pretty soon you'll need more and
will either need to design some Turing complete language with Mickey
Mouse syntax or have things punted to pre-made procedures in a real
language through indirection tags with parameters [that essentially
are procedure calls]. I'm sure you're aware that this all has been
tried (Zope's zhtml, aolserver's adp etc.).
(I am not mentioning xslt and such here because of your footnote _and_
because they don't fit the particular problem you have characterized).
What would be a lot more interesting is inferring a substantial
part of the 'look and feel' of a site from existing pages. When
the client calls you for some dynamic addition to a site that already has
been designed by html guys, can you run a spider on the site and make sure
that your tags (for example) generate the right _html_ for the site by using
the inferred style? CSS helps you to an extent there, but not everyone
uses it and it might not go far enough. I don't know if there's anything
out there that attempts to do this.
cheers,
BM
Dr. Edmund Weitz
> I am pretty sure CL-HTTP has this functionality. The server expands
> special tags into Lisp code. Not sure if it compiles it or not.
> The web site does not seem to working correctly at the moment.
I'd love to play around with CL-HTTP a bit, but I'm somewhat kept from
it by the almost complete lack of documentation (or at least I didn't
find it). Also, I was told that the license is not very clear so I
wouldn't know if I could use it for commercial work.
> What you are asking is not that difficult. Just suck everything
> which is not the Lisp tag into a string until you hit the tag, then
> the code is inserted.
I didn't say it was difficult, in fact I have implemented it
already... :)
> By why just insert the state of a var, why not insert Lisp code?
Because that would violate the whole idea of separating code and HTML
as far as possible. See the earlier paragraphs in the posting that you
replied to.
> The main limitation with this solution, for example, is having the
> Lisp code insert a variable number of rows into a table. The code
> needs facilities to generate tags.
Yeah, you obviously didn't check the link to HTML::Template that I
provided. (I said that my example was a simplification of the
process.)
HTML::Template gets around with basically only three tags, TMPL_VAR,
TMPL_LOOP and TMPL_IF. I've used this module for more than two years
now and I've found that it has enough expressive power to generate
every complex HTML that I need. It is also plain easy at the same time
and doesn't get in the way of the graphic designer - exactly what I
described as my goal in my posting.
Cheers,
Edi.
Vebjorn Ljosa
|
| There do exist a couple of very sophisticated libraries like onShore's
| IMHO or AllegroServe's HTML generation facility, to name only two, but
| (at least for my situation) they seem to have the wrong approach: They
| put the task of generating the complete HTML page into the
| program. Almost every project that I was involved in had one or more
| graphic designers who were responsible for spewing out the HTML while
| developers like me used these templates and added the application
| logic, so putting the HTML into the program, whether Lisp or not,
| would not be an option.
|
| The other extreme is to embed the programming language into the HTML
| like with ASP, JSP, PHP, and so on. From my experience this leads to
| code that is very hard to read and maintain and apart from that is
| also simply unesthetic.[2]
|
| The approach that worked best for me is to separate HTML and
| application as far as possible and leave only a minimum of 'code' in
| the HTML template, so when I could choose which approach to use I
| usually opted for Sam Tregar's HTML::Template
| <http://sourceforge.net/projects/html-template/>. (Forget for a
| moment, if you can, that it's based on Perl... :)
Another approach is that of Webmacro <URL:http://www.webmacro.org/>, a
template system for Java. It has a simple and terse, but very
limited, language for use in templates: in addition to simple
conditionals, you have the ability to print out the value of a
variable and loop over the items of a collection. The language is
simple enough that a web designer can learn it in a few minutes, and
limited enough that nobody's tempted to put actual code in the middle
of a template. Here's an example of a Webmacro template:
#set $Response.ContentType = "text/html"
<HTML><HEAD><TITLE>$Customer.Name</TITLE></HEAD><BODY bgcolor='white'>
<h1>$Customer.Name: History<h1>
Here's a list of your orders since $Customer.Orders.StartDate:
<table width='70%'>
<th><td>Order Date</td><td>Item Requested</td>
<td>Number of Units</td></th>
#foreach $order in $Customer.Orders {
<tr>
<td>$order.Date</td> <td>$order.Item.Name</td> <td>$order.Number</td>
</tr>
}
</table> </body> </html>
This way of doing things allows a pleasant division of labor between
the programmer and the web designer: the programmer gets the
information that needs to be presented on the page, but the web
designer decides which piece goes where on the web page.
I used Webmacro for a small project a year or so ago, and was very
happy with it. Maybe something similar for Lisp could be useful.
--
Vebjorn Ljosa
Dr. Edmund Weitz
> Yes, but a Wade Humeniuk. pointed out pretty soon you'll need more
> and will either need to design some Turing complete language with
> Mickey Mouse syntax or have things punted to pre-made procedures in
> a real language through indirection tags with parameters [that
> essentially are procedure calls].
See my response to Wade Humeniuk's post and the reference for
HTML::Template at <http://sourceforge.net/projects/html-template/>. It
has two more tags that make it, er, let's say "Turing complete
enough". I've been using it successfully for more than two years and
have never missed a feature. It's still easy enough not to interfer
with the graphic designers and the bloated tools they sometimes use -
like Adobe GoLive, Macromedia Dreamweaver, you name it.[1]
Let me add that very often you have a two-way process: Not only are
you given pages done by graphic designers to "fill in the blanks", but
the designers will also in turn have to work on your "coded" pages if
the client wants/needs changes in the page layout. So they'd better be
able to deal with what you've written into their pages. In an ideal
world they wouldn't see your stuff, the application logic, at all.
> I'm sure you're aware that this all has been tried (Zope's zhtml,
> aolserver's adp etc.).
Sure. I've seen and tried most of the variants and for me they all
suffer from the same limitations.
> What would be a lot more interesting is inferring a substantial part
> of the 'look and feel' of a site from existing pages. When the
> client calls you for some dynamic addition to a site that already
> has been designed by html guys, can you run a spider on the site and
> make sure that your tags (for example) generate the right _html_ for
> the site by using the inferred style? CSS helps you to an extent
> there, but not everyone uses it and it might not go far enough. I
> don't know if there's anything out there that attempts to do this.
I'm not aware of such thing but it would sure be nice. It sounds like
a way to _completely_, not only partially, separate HTML from program
logic. But I think this would require a fair amount of intelligence
from our imaginary tool.
At the end of the day you probably still have to give hints as to what
has to be taken literally and what is a template. You'll also have to
say if you're template is to be filled out once or if it's a template
that has to be repeated a couple of times. Add logical alternatives
(IF/THEN) to this and you end up with exactly the three tags that
HTML::Template provides... :)
Edi.
[1] On a completely unrelated side-note: It is my experience that the
HTML page designers that are really good don't use these tools but
rather work with editors like UltraEdit, BBEdit, or Emacs. Not
really surprising, I think.
Dr. Edmund Weitz
I've heard of Webmacro but didn't try it before. From your description
it sounds pretty similar to HTML::Template (I wonder which one was
first...) and I'll check it out to see if it has something interesting
that HTML::Template hasn't.
I completely agree with your remarks about the division of labor. This
reflects my professional situation and this was what I tried to
explain in my original posting.
Thanks,
Edi.
Wade Humeniuk
"Vebjorn Ljosa" <ab...@ljosa.com> wrote in message
news:cy34rkz...@ljosa.com...
> template system for Java. It has a simple and terse, but very
> limited, language for use in templates: in addition to simple
> conditionals, you have the ability to print out the value of a
> variable and loop over the items of a collection. The language is
> simple enough that a web designer can learn it in a few minutes, and
> limited enough that nobody's tempted to put actual code in the middle
> of a template. Here's an example of a Webmacro template:
>
This is something that is an issue for me. The insistence that
"non-programmers", in this case a Web Designer, should not be expoused to
"code". Much of the grief that goes on is the attempt to hide that someone
is actually programming. What good is it for someone to be able to learn a
language in a "few minutes"? All that tends to happen is that when someone
wants to something more complex, with that simple language, is that they end
up creating reams of non-sensical work for themselves.
Even using the core HTML, people are programming. It just that the program
is spread over multiple files, the conditions being hyperlinks. Look how
fast Javascript was needed, then applets and every kind of plugin. Then all
those tools hiding the fact that people were programming. Lots of denial.
I am not saying that only programming in a text editor is good but that
shying away from the fact (that it is all programming) is bad. There now
becomes "programmers" and "non-programmers". Instead of people creating
computer programs in a common language, with strict syntax and semantics,
and unambiguous meaning.
> #set $Response.ContentType = "text/html"
> <HTML><HEAD><TITLE>$Customer.Name</TITLE></HEAD><BODY bgcolor='white'>
> <h1>$Customer.Name: History<h1>
>
> Here's a list of your orders since $Customer.Orders.StartDate:
> <table width='70%'>
> <th><td>Order Date</td><td>Item Requested</td>
> <td>Number of Units</td></th>
> #foreach $order in $Customer.Orders {
> <tr>
> <td>$order.Date</td> <td>$order.Item.Name</td>
<td>$order.Number</td>
> </tr>
> }
> </table> </body> </html>
>
Look like TCL imbedded in HTML.
Joe Arnold
e...@agharta.de (Dr. Edmund Weitz) wrote:
> I've heard of Webmacro but didn't try it before. From your description
> it sounds pretty similar to HTML::Template (I wonder which one was
> first...) and I'll check it out to see if it has something interesting
> that HTML::Template hasn't.
I have been with happy and productive with BRL. It uses Kawa (a Scheme
written in Java) and servlets. There are good examples at
<URL:http://brl.sourceforge.net/>.
Dennis
Thomas F. Burdick
> Bulent Murtezaoglu <b...@acm.org> writes:
>
> > Yes, but a Wade Humeniuk. pointed out pretty soon you'll need more
> > and will either need to design some Turing complete language with
> > Mickey Mouse syntax or have things punted to pre-made procedures in
> > a real language through indirection tags with parameters [that
> > essentially are procedure calls].
>
> See my response to Wade Humeniuk's post and the reference for
> HTML::Template at <http://sourceforge.net/projects/html-template/>. It
> has two more tags that make it, er, let's say "Turing complete
> enough". I've been using it successfully for more than two years and
> have never missed a feature. It's still easy enough not to interfer
> with the graphic designers and the bloated tools they sometimes use -
> like Adobe GoLive, Macromedia Dreamweaver, you name it.[1]
>
> Let me add that very often you have a two-way process: Not only are
> you given pages done by graphic designers to "fill in the blanks", but
> the designers will also in turn have to work on your "coded" pages if
> the client wants/needs changes in the page layout. So they'd better be
> able to deal with what you've written into their pages. In an ideal
> world they wouldn't see your stuff, the application logic, at all.
This has, in the past, been an important criterion for me as well.
The half-way point of making the designers learn a simple template
system, and making the coders use it, worked pretty well. As for a
solution to "here we put a table with varying numbers of rows and
columns" ... well, no one siad you need to generate a single page from
a single template file. In such a situation, I've mocked up a simple
page that shows the functionality, and breaks things apart into
templates for reasonable widgets. Designers can deal with that just
fine (although I wouldn't trust them to do it themselves). You can
then go along building and inserting widgets to your heart's content,
using nothing but simple replacement.
--
/|_ .-----------------------.
,' .\ / | No to Imperialist war |
,--' _,' | Wage class war! |
/ / `-----------------------'
( -. |
| ) |
(`-. '--.)
`. )----'
Dr. Edmund Weitz
> This is something that is an issue for me. The insistence that
> "non-programmers", in this case a Web Designer, should not be
> expoused to "code". Much of the grief that goes on is the attempt
> to hide that someone is actually programming. What good is it for
> someone to be able to learn a language in a "few minutes"? All that
> tends to happen is that when someone wants to something more
> complex, with that simple language, is that they end up creating
> reams of non-sensical work for themselves.
>
> Even using the core HTML, people are programming. It just that the
> program is spread over multiple files, the conditions being
> hyperlinks. Look how fast Javascript was needed, then applets and
> every kind of plugin. Then all those tools hiding the fact that
> people were programming. Lots of denial. I am not saying that only
> programming in a text editor is good but that shying away from the
> fact (that it is all programming) is bad. There now becomes
> "programmers" and "non-programmers". Instead of people creating
> computer programs in a common language, with strict syntax and
> semantics, and unambiguous meaning.
In an ideal world I would heartily agree with you. In fact, whenever
one of my friends asks me which tool he or she should use for HTML
generation, I suggest they should use a plain text editor and read a
good book or online tutorial until they've understood the basics. I
suggest they should not start to use one of the fancy "WYSIWYG" HTML
editors until they know what they're doing.
In my ideal world people would also refrain from using those new
digital gizmo cameras and use an old Leica rangefinder camera to
really get a feeling of what they're doing. Everybody would invest
some time in learning to use RPN pocket calculators and see that
they're really superior to "normal" ones. Most of the time they
wouldn't need those calculators, though, because everybody knows how
to use his slide rule and is pretty good at mental arithmetic. And of
course I would be able to fix my car or my washing machine if they're
broken... :)
Unfortunately, the world I live in is not that ideal. I sometimes have
to work for money because my wife and my daughter want something to
eat, and thus I have to cope with graphic designers who simply don't
want to learn a programming language. They long for the good old days
when everything they did ended up on paper and looked more or less
exactly like they wanted it to be. They started with HTML because
somebody told them this would be the next big thing, but actually they
want to use Quark XPress and nothing else. For want of an alternative
they want "WYSIWYG" HTML editors that act as if they _were_ Quark
XPress. They don't care about HTML, let alone Javascript or something
more sophisticated, or did you think they knew the slightest bit about
Postscript back in the "paper age"?[1]
Cheers,
Edi.
[1] This is of course grossly generalizing and unfair to say the
least. I just wanted to re-iterate that I started this whole
discussion based on the premise that I was searching a pragmatic
solution for a real-world problem. We can change it to a more
philosophical viewpoint and I'm sure I'll mostly agree with you.
Dr. Edmund Weitz
> This has, in the past, been an important criterion for me as well.
> The half-way point of making the designers learn a simple template
> system, and making the coders use it, worked pretty well. As for a
> solution to "here we put a table with varying numbers of rows and
> columns" ... well, no one siad you need to generate a single page
> from a single template file. In such a situation, I've mocked up a
> simple page that shows the functionality, and breaks things apart
> into templates for reasonable widgets. Designers can deal with that
> just fine (although I wouldn't trust them to do it themselves). You
> can then go along building and inserting widgets to your heart's
> content, using nothing but simple replacement.
This also sound like a viable solution to me.
Thanks,
Edi.
Marc Spitzer
Have you looked at xmlc (http://xmlc.enhydra.org/index.html)? If I
remember correctly it completely separates the html from the
application logic. The html author and the java programmer get together
and agree on names( class="name") for what the programmer needs to do.
Then the xmlc compiler creates a servlet and builds a dom where the
programmer fills in the blanks, quite cool.
marc
Dr. Edmund Weitz
I've had a look at BRL before and for me it suffers from the same
problems as PHP, ASP, JSP or Embperl (apart from the fact that it
maybe has a nicer syntax): It'll enable you to write whole programs
inside of HTML pages which is exactly what I don't want. See my
previous postings in this thread.
Thanks,
Edi.
Dr. Edmund Weitz
> Have you looked at xmlc (http://xmlc.enhydra.org/index.html)? If I
> remember correctly it completely separates the html from the
> application logic. The html author and the java programmer get
> together and agree on names( class="name") for what the programmer
> needs to do. Then the xmlc compiler creates a servlet and builds a
> dom where the programmer fills in the blanks, quite cool.
Thanks for the link! I didn't know XMLC before and it looks very
interesting. It also looks much harder to implement than the simple
HTML::Template stuff, though, but it nevertheless is an interesting
approach.
Thanks again,
Edi.
Marc Battyani
"Dr. Edmund Weitz" <e...@agharta.de> wrote
> I have to cope with graphic designers who simply don't
> want to learn a programming language. They long for the good old days
> when everything they did ended up on paper and looked more or less
> exactly like they wanted it to be. They started with HTML because
> somebody told them this would be the next big thing, but actually they
> want to use Quark XPress and nothing else. For want of an alternative
> they want "WYSIWYG" HTML editors that act as if they _were_ Quark
> XPress. They don't care about HTML, let alone Javascript or something
> more sophisticated, or did you think they knew the slightest bit about
> Postscript back in the "paper age"?[1]
Tell them to use CL-PDF instead... :))
Sorry couldn't resist... BTW I've had the same kind of problems and this is
why I made mod_lisp, so that I can tell managers : Look, it's just an Apache
server + a few open source modules.
Marc
PS: I have a RPN calculator...
Dave MacDonald
Well, I couldn't resist chiming in. Here's a link to an interesting
idea that should be generalizable to a Lisp-oriented web-publishing
framework (it's already implemented in the Zope web framework):
http://www.zope.org/Documentation/Articles/ZPT1
I've never used it, but it seems meet your criteria for completely
separating HTML from web programming.
-Dave
Marc Spitzer
I don't know, the hardest part is creating the dom tree and the 'stuff'
to modify it as needed. Most of that should be laying around some
where on the internet. But let me say I do not do estimates well and I
am not very familiar with the problem.
Here is another link opensource.go.com they did a "little language"
for page generation inside of servlets. Its like webmacro, but it
does a more complete job(I think anyway).
marc
Dr. Edmund Weitz
> Sorry couldn't resist... BTW I've had the same kind of problems and
> this is why I made mod_lisp, so that I can tell managers : Look,
> it's just an Apache server + a few open source modules.
I know mod_lisp and I like it. I have used it a couple of times. The
small template engine that I'm working on will be webserver-agnostic,
i.e. it will work with mod_lisp as well as with (Portable)
AllegroServe, CL-HTTP, Araneida or even from a (CL) CGI script.
> PS: I have a RPN calculator...
That's good. Me too, of course... :)
Marc Spitzer
Well I have had an hp 11c & 48s but now all I have is bc, does that count?
marc
Ed L Cashin
Only literally. dc would count figuratively too, being an RPN
calculator. :)
--
--Ed L Cashin | PGP public key:
eca...@uga.edu | http://noserose.net/e/pgp/
Brian P Templeton
And I've got dc and several Forth implementations... :)
> --
> --Ed L Cashin | PGP public key:
> eca...@uga.edu | http://noserose.net/e/pgp/
--
BPT <b...@tunes.org> /"\ ASCII Ribbon Campaign
backronym for Linux: \ / No HTML or RTF in mail
Linux Is Not Unix X No MS-Word in mail
Meme plague ;) ---------> / \ Respect Open Standards
Brian P Templeton
^^^^^^^^^^^^^^^^^
Why are you getting graphic designers to create your WWW pages?
> want to learn a programming language. They long for the good old days
> when everything they did ended up on paper and looked more or less
> exactly like they wanted it to be. They started with HTML because
> somebody told them this would be the next big thing, but actually they
> want to use Quark XPress and nothing else. For want of an alternative
> they want "WYSIWYG" HTML editors that act as if they _were_ Quark
> XPress. They don't care about HTML, let alone Javascript or something
> more sophisticated, or did you think they knew the slightest bit about
> Postscript back in the "paper age"?[1]
>
> Cheers,
> Edi.
>
> [1] This is of course grossly generalizing and unfair to say the
> least. I just wanted to re-iterate that I started this whole
> discussion based on the premise that I was searching a pragmatic
> solution for a real-world problem. We can change it to a more
> philosophical viewpoint and I'm sure I'll mostly agree with you.
--
Brian P Templeton
Well said! I agree - false divisions between `programmers' and
`non-programmers' and syntax that's complex^W``user friendly'' enough
to cause cancer of the semi-colon don't appear to make WWW site
creation /easier/... or, for that matter, neither does users using
things like MS-Word to create documents without bothering to
understand the basics of typesetting (instead, they use their ``word
processor'' like a typewriter with extra typefaces), and depending on
``easy to use'', ``integrated'' features (where ``integrated'' just
means it's in the same process) instead of understanding how to use
their ``editor''...
>
>> #set $Response.ContentType = "text/html"
>> <HTML><HEAD><TITLE>$Customer.Name</TITLE></HEAD><BODY bgcolor='white'>
>> <h1>$Customer.Name: History<h1>
>>
>> Here's a list of your orders since $Customer.Orders.StartDate:
>> <table width='70%'>
>> <th><td>Order Date</td><td>Item Requested</td>
>> <td>Number of Units</td></th>
>> #foreach $order in $Customer.Orders {
>> <tr>
>> <td>$order.Date</td> <td>$order.Item.Name</td>
> <td>$order.Number</td>
>> </tr>
>> }
>> </table> </body> </html>
>>
>
> Look like TCL imbedded in HTML.
>
Or (yuck) Perl. Of course, if it were Perl, the author would use a
regular expression to hopefully extract the correct data and item name
out of a large string :).
Reini Urban
>http://www.zope.org/Documentation/Articles/ZPT1
>I've never used it, but it seems meet your criteria for completely
>separating HTML from web programming.
I've tried TAL and its extensions last summer, but was not convinced
enough to buy it. We just lost one month of paid work. But better than
DTML at least.
The syntax is okay for the template seperation goal:
<dtml-var expr="python:int(_['x']) + _['y']">
=> <lisp expr="(+ (coerce (global x) 'integer) (global y))" />
--
Reini Urban
http://xarch.tu-graz.ac.at/autocad/news/faq/autolisp.html