reading/writing bytes smaller than 8 bits?
Bruce L. Lambert
When I write n (unsigned-byte 4)s to my file-system (solaris 2.6 on a sparc
clone), and then I say ls -l to look at the size of the file, there are n
bytes in the file rather than n/2 as I would have expected. Does this mean
the file system can write nothing smaller than an 8-bit byte? If so, does
this in any way affect what happens when I read an (unsigned-byte 4) from
the same file. I've played with this some, and the values appear to be
correct, but I'm still wondering what's happening 'below the surface'. Is it
still the case that an array of n 4-bit bytes will occupy (roughly) n/2
bytes of main memory when loaded into lisp? (Apologies if this is an
elementary computer science question.)
Thanks.
--
Bruce L. Lambert
lamb...@uic.edu
Barry Margolin
How they're represented in memory is an implementation detail. When you
declare an array specialized to (unsigned-byte 4), you're saying that the
array elements have to be able to hold at least 0-15, but there's nothing
preventing the implementation from using larger elements. You're simply
promising that you won't try to store anything larger, and the
implementation can make any optimizations it wants based on it, but isn't
required to do so. The UPGRADED-ARRAY-ELEMENT-TYPE function should tell
you what type it actually uses.
As far as file I/O goes, all that's guaranteed is that if you write a file
with a particular element type, and then read it back with the same element
type, you'll get the same values. It would be difficult to specify how the
data is represented in the file in an OS-independent way. For instance,
how should it write (unsigned-byte 8) elements on a machine with with
36-bit words and 9-bit bytes? Should it write each 8-bit byte into a 9-bit
file byte or pack them in tightly? If it packs them, should the 5th byte
be written to the 2nd word or straddle the word boundaries? Should it be
big-endian or little-endian?
--
Barry Margolin, bar...@bbnplanet.com
GTE Internetworking, Powered by BBN, Burlington, MA
*** DON'T SEND TECHNICAL QUESTIONS DIRECTLY TO ME, post them to newsgroups.
Please DON'T copy followups to me -- I'll assume it wasn't posted to the group.
Erik Naggum
| When I write n (unsigned-byte 4)s to my file-system (solaris 2.6 on a
| sparc clone), and then I say ls -l to look at the size of the file, there
| are n bytes in the file rather than n/2 as I would have expected.
why did you expect that?
| Does this mean the file system can write nothing smaller than an 8-bit
| byte?
the file system has nothing to do with it. Unix doesn't know about file
contents at all, and as an operating system design, takes significant
pride in that fact.
| If so, does this in any way affect what happens when I read an
| (unsigned-byte 4) from the same file.
if you write (unsigned-byte 4) to a file and read it again, you should
get the exact same values back. how these bits are stored is none of
your business.
if your expectations can be defended and you have reason to use this
feature to obtain improved storage performance, talk with your CL vendor
and have them make it work for you.
| I've played with this some, and the values appear to be correct, but I'm
| still wondering what's happening 'below the surface'.
the only people who can really answer that question is the person who
implemented the feature in your Common Lisp implementation. finding that
person may be hard, but if you use free software, the source is there.
if not, you should use the customer support facilities of the vendor.
| Is it still the case that an array of n 4-bit bytes will occupy (roughly)
| n/2 bytes of main memory when loaded into lisp?
if it makes performance sense to do so, or people have argued for this
feature even if it didn't, it will. you can, however, easily measure the
space costs of an allocation. for instance,
(time (make-array 1000 :element-type '(unsigned-byte 4) :initial-element 0))
; cpu time (non-gc) 0 msec user, 0 msec system
; cpu time (gc) 0 msec user, 0 msec system
; cpu time (total) 0 msec user, 0 msec system
; real time 14 msec
; space allocation:
; 9 cons cells, 0 symbols, 512 other bytes, 0 static bytes
=> #(0 0 0 0 0 0 0 0 0 0 ...)
(4) cl-user
(time (make-array 1000 :element-type '(unsigned-byte 8) :initial-element 0))
; cpu time (non-gc) 0 msec user, 0 msec system
; cpu time (gc) 0 msec user, 0 msec system
; cpu time (total) 0 msec user, 0 msec system
; real time 0 msec
; space allocation:
; 9 cons cells, 0 symbols, 1,016 other bytes, 0 static bytes
=> #(0 0 0 0 0 0 0 0 0 0 ...)
(5) cl-user
(time (make-array 1000 :element-type '(unsigned-byte 2) :initial-element 0))
; cpu time (non-gc) 0 msec user, 0 msec system
; cpu time (gc) 0 msec user, 0 msec system
; cpu time (total) 0 msec user, 0 msec system
; real time 0 msec
; space allocation:
; 9 cons cells, 0 symbols, 512 other bytes, 0 static bytes
=> #(0 0 0 0 0 0 0 0 0 0 ...)
(6) cl-user
(time (make-array 1000 :element-type '(unsigned-byte 1) :initial-element 0))
; cpu time (non-gc) 0 msec user, 0 msec system
; cpu time (gc) 0 msec user, 0 msec system
; cpu time (total) 0 msec user, 0 msec system
; real time 0 msec
; space allocation:
; 9 cons cells, 0 symbols, 144 other bytes, 0 static bytes
=> #*0000000000.. ; [abbreviated by me]
(7) cl-user
(time (make-array 1000 :element-type '(unsigned-byte 16) :initial-element 0))
; cpu time (non-gc) 0 msec user, 0 msec system
; cpu time (gc) 0 msec user, 0 msec system
; cpu time (total) 0 msec user, 0 msec system
; real time 0 msec
; space allocation:
; 9 cons cells, 0 symbols, 2,016 other bytes, 0 static bytes
=> #(0 0 0 0 0 0 0 0 0 0 ...)
the reported results show the storage strategies and optimizations for
various data types.
#:Erik
Bruce L. Lambert
Thanks for your response.
Erik Naggum <er...@naggum.no> wrote in message
news:<31578251...@naggum.no>...
> * "Bruce L. Lambert" <lamb...@uic.edu>
> | When I write n (unsigned-byte 4)s to my file-system (solaris 2.6 on a
> | sparc clone), and then I say ls -l to look at the size of the file,
there
> | are n bytes in the file rather than n/2 as I would have expected.
>
> why did you expect that?
I just figured 1 byte = 8 bits, therefore 1 (unsigned-byte 4) = 4 bits = 0.5
bytes both in Lisp and in a file on disk. Simple, yet erroneous, deductive
logic on my part.
> | Does this mean the file system can write nothing smaller than an 8-bit
> | byte?
>
> the file system has nothing to do with it. Unix doesn't know about file
> contents at all, and as an operating system design, takes significant
> pride in that fact.
If not the OS, then what system determines how an (unsigned-byte 4) or any
other object gets written to disk? Does each application make its own
decisions on this point?
> | If so, does this in any way affect what happens when I read an
> | (unsigned-byte 4) from the same file.
>
> if you write (unsigned-byte 4) to a file and read it again, you should
> get the exact same values back. how these bits are stored is none of
> your business.
>
> if your expectations can be defended and you have reason to use this
> feature to obtain improved storage performance, talk with your CL vendor
> and have them make it work for you.
I just was trying to understand the space considerations of some code I'm
writing. I figured there was a direct mapping from the size of the internal
data structure to the size of the file. That is, if a thousand-element array
of 4-bit bytes takes 512 bytes (according to time), and I write the contents
of that array to disk, I expected to see a 512 byte file. Not so,
apparently.
> | I've played with this some, and the values appear to be correct, but I'm
> | still wondering what's happening 'below the surface'.
>
> the only people who can really answer that question is the person who
> implemented the feature in your Common Lisp implementation. finding
that
> person may be hard, but if you use free software, the source is there.
> if not, you should use the customer support facilities of the vendor.
Allegro is my vendor. Duane Rettig could probably answer this question if he
had the inclination, but my need to know is not that urgent. I just wanted
to understand what was going on. You have helped me do that. (Although the
depths of my misunderstanding about these kinds of issues cannot possibly be
dealt with in one message---that's a separate problem.)
-bruce
Robert Monfera
Hi Bruce,
If it is a solid requirement that 4 bits' worth of data occupy no more
than 4 bits in the file, and the smallest unit you can write is 8 bits,
then a practical way to do it is to merge 4-bit values into 8-bit values
before writing (and vice versa after reading). Check out the Numbers
section of the Hyperspec for bit-manipulation like ASL.
If it is not a strict requirement, don't worry about the overhead or use
ZIP :-)
Robert
"Bruce L. Lambert" wrote:
> When I write n (unsigned-byte 4)s [...]
Johan Kullstam
> Erik,
>
> Thanks for your response.
>
> Erik Naggum <er...@naggum.no> wrote in message
> news:<31578251...@naggum.no>...
> > * "Bruce L. Lambert" <lamb...@uic.edu>
> > | When I write n (unsigned-byte 4)s to my file-system (solaris 2.6 on a
> > | sparc clone), and then I say ls -l to look at the size of the file,
> there
> > | are n bytes in the file rather than n/2 as I would have expected.
> >
> > why did you expect that?
>
>
> I just figured 1 byte = 8 bits, therefore 1 (unsigned-byte 4) = 4 bits = 0.5
> bytes both in Lisp and in a file on disk. Simple, yet erroneous, deductive
> logic on my part.
the system likes to pack things in 8-bit bytes. you seem to want to
save space. therefore, pack your 4 bit objects into 8 bit ones. keep
in mind that this may be a useless optimization. you must decide if
space is all that important.
LDB and DPB are your friends.
i offer you a couple functions i wrote a while back. please excuse
the short variable names. i was fighting unix/C brain-damage at the
time. please feel free to use them as you wish.
;; pack bytes into an integer
(defun pack-word (bl ; byte list
sl ; size list
&optional
(acc 0) ; original byte
(pos 0)) ; initial position
(if (and bl sl)
(pack-word (cdr bl)
(cdr sl)
(dpb (car bl) (byte (car sl) pos) acc)
(+ pos (car sl)))
(values acc pos)))
;; unpack bytes in integer word word using lengths in the size list
(defun unpack-word (word ; packed word
sl) ; size list
(let ((pos 0)
(bl nil))
(dolist (sz sl)
(push (ldb (byte sz pos) word) bl)
(incf pos sz))
(values (nreverse bl) pos)))
examples:
USER(18): (pack-word '(3 9) '(4 4))
147
8
USER(19): (unpack-word 147 '(4 4))
(3 9)
8
hope this helps.
--
J o h a n K u l l s t a m
[kull...@ne.mediaone.net]
Don't Fear the Penguin!
Andrew Cooke
Barry Margolin <bar...@bbnplanet.com> wrote:
> As far as file I/O goes, all that's guaranteed is that if you write a
file
> with a particular element type, and then read it back with the same
element
> type, you'll get the same values. It would be difficult to specify
how the
> data is represented in the file in an OS-independent way. For
instance,
> how should it write (unsigned-byte 8) elements on a machine with with
> 36-bit words and 9-bit bytes? Should it write each 8-bit byte into a
9-bit
> file byte or pack them in tightly? If it packs them, should the 5th
byte
> be written to the 2nd word or straddle the word boundaries? Should it
be
> big-endian or little-endian?
Ouch! Next week I'm planning to write code that generates binary files
to a certain standard (Midi music data). I haven't checked the docs
yet, but this post implies that there is no way to generate files with a
specified binary format (at least that other people or other Lisps can
use with certainty). Is that correct? What do people do when they need
to write a formatted binary file? Or does everyone use just Lisp
programs ;-) ?
Andrew
Sent via Deja.com http://www.deja.com/
Before you buy.
Duane Rettig
> Allegro is my vendor. Duane Rettig could probably answer this question if he
> had the inclination, but my need to know is not that urgent. I just wanted
> to understand what was going on. You have helped me do that. (Although the
> depths of my misunderstanding about these kinds of issues cannot possibly be
> dealt with in one message---that's a separate problem.)
Well, if it becomes urgent, do write to the bug list at Franz; it has
been quite a while since your last spr ...
I don't want to get into details about the specific problem that
was being discussed, but I do want to respond to this:
> I just figured 1 byte = 8 bits, therefore 1 (unsigned-byte 4) = 4 bits = 0.5
> bytes both in Lisp and in a file on disk. Simple, yet erroneous, deductive
> logic on my part.
A mistake made by many. Someday we who are over 40 will die off or
go to pasture and a byte may then become redefined as 8 bits, though
CL would have to be redefined in order for that to occur. When
it was suggested that we at Franz speed up this process, I did some
research and cam up with this answer (somewhat edited):
========
See http://t2r.uwasa.fi/jargon/byte.html for a fairly thorough discussion
of the term "byte". It generally follows my own understanding of the term,
as well as the info I was able to search for on the net. As a former
Sperry/Univac employee, I am knowledgable of bytes being anywhere from
5 to 9 bits, especially on 36-bit machines (the Univac 1100 series
had these different byte sizes). But I hadn't known about the PDP-10,
whose byte sizes could vary from 1 to 36 bits! ... It was the
aforementioned variability of the byte on which CL based its term ...
[ ... ]
IBM (really Werner Buchholz) coined the term byte in 1956. But until
the 70's or 80's, it was never clear what size you were talking about,
unless you were clear on your context: if you were in IBM culture,
the byte was clearly 8 bits most of the time. But if you were in the
company of Eniac/Univac or DEC people, then a byte could mean simply
"a character" which also implied many potential sizes. It is ironic
that we are actually departing from the old tradition of calling a
byte a character, by going to two-byte characters. [ed: but not always
two bytes]
I would guess that whatever committee [a developer] said standardized on
the term "octet" was aware of this issue, and I wouldn't be surprised
if their reasoning for letting go of "byte" wasn't precisely this
reason that it would be too non-definitive.
======
Because of this and further discussion, we have standardized on the
term "octet" when we describe 8-bits specifically. The proposal that
resulted from this is part of a design of an I/O system, which will
complement the current Gray streams system we currently have, but
which will address various issues of stream mutivalency, multibyte
character sets and locales, speed, and a few other problems that the
CL spec and Gray streams don't currently address. We hope to publish
this proposal soon, long before our next major release.
--
Duane Rettig Franz Inc. http://www.franz.com/ (www)
1995 University Ave Suite 275 Berkeley, CA 94704
Phone: (510) 548-3600; FAX: (510) 548-8253 du...@Franz.COM (internet)
Frode Vatvedt Fjeld
> I haven't checked the docs yet, but this post implies that there is
> no way to generate files with a specified binary format (at least
> that other people or other Lisps can use with certainty). Is that
> correct?
It may not be strictly required by a CL implementation, but I think
it's a reasonably safe bet that the element-type (unsigned-byte 8) is
stored as octets on current unix-type systems.
> What do people do when they need to write a formatted binary file?
> Or does everyone use just Lisp programs ;-) ?
You may want to have a look at this little package I wrote for reading
and writing "binary" files:
<URL:http://www.cs.uit.no/~frodef/sw/binary-types/>
--
Frode Vatvedt Fjeld
Marco Antoniotti
Duane Rettig <du...@franz.com> writes:
...
>
> Because of this and further discussion, we have standardized on the
> term "octet" when we describe 8-bits specifically. The proposal that
> resulted from this is part of a design of an I/O system, which will
> complement the current Gray streams system we currently have, but
> which will address various issues of stream mutivalency, multibyte
> character sets and locales, speed, and a few other problems that the
> CL spec and Gray streams don't currently address. We hope to publish
> this proposal soon, long before our next major release.
>
Is this a X3J13 proposal or a Franz proprietary one?
Cheers
--
Marco Antoniotti ===========================================
PARADES, Via San Pantaleo 66, I-00186 Rome, ITALY
tel. +39 - 06 68 10 03 17, fax. +39 - 06 68 80 79 26
http://www.parades.rm.cnr.it/~marcoxa
Erik Naggum
| Is this a X3J13 proposal or a Franz proprietary one?
how can _proposals_ be proprietary? sheesh.
work on this topic was "commissioned" at the last X3J13 meeting, and
Franz accepted the responsibility to oversee the work if not write it.
#:Erik
Erik Naggum
| Ouch! Next week I'm planning to write code that generates binary files
| but this post implies that there is no way to generate files with a
| specified binary format (at least that other people or other Lisps can
| to write a formatted binary file? Or does everyone use just Lisp
| programs ;-) ?
locate your misplaced calm and _breathe_, mister, you're near hysterical.
if you know that the medium is an 8-bit medium, opening a stream for
writing to or reading from it as (UNSIGNED-BYTE 8) is guaranteed to
produce the expected results by virtue of the insanity of not doing so.
if you need control over the individual bits, you don't request a service
where you explicitly say "do the smartest thing you can with my request".
please realize that the Common Lisp community is more than 40 years old.
collectively, the community has already been where every clueless newbie
will be going for the next three years. so relax, please.
#:Erik
Erik Naggum
| 0.5 bytes both in Lisp and in a file on disk. Simple, yet erroneous,
| deductive logic on my part.
byte n. 1. adjacent bits within an _integer_. (The specific number of bits
can vary from point to point in the program; see the _function_ BYTE.)
-- from the Common Lisp the Standard (ANSI X3.226-1994)
| If not the OS, then what system determines how an (unsigned-byte 4) or
| any other object gets written to disk? Does each application make its
| own decisions on this point?
yes. Unix doesn't know about file contents. it stores only bytes (still
of no predetermined size). the only common byte size these days is 8,
but Unix was delivered in the past on machines with 9-bit bytes. (this
historical fact has even escaped most C programmers.)
| I just was trying to understand the space considerations of some code I'm
| writing. I figured there was a direct mapping from the size of the
| internal data structure to the size of the file.
there never are.
the first tenet of information representation is that external and
internal data formats are incommensurate concepts. there simply is no
possible way they could be conflated conceptually. to move from external
to internal representation, you have to go through a process of _reading_
the data, and to move from internal to external representation, you have
to go through a process of _writing_ the data. these processes are
non-trivial, programmatically, conceptually, and physically. that some
languages make them appear simple is to their credit, but as always,
programming languages are about _pragmatics_, and it doesn't make sense
to make conceptually complex things complex to use and program -- quite
the contrary, in fact. so the more things that are different look the
same to a programmer, the higher the likelihood that there is something
complex and worthwhile going on behind the scenes.
the second tenet of information representation is that optimizations for
internal representations are invalid for external representation and vice
versa. the crucial difference is that data in internal representation is
always under the control of the exact same software at all times, while
data in external representation _never_ is. therefore, decisions you
make when optimizing internal representation exploit the fact that you
have full control over the resources that are being used to represent it,
such as actually knowing all the assumptions that might be involved,
while decisions you make when optimizing external representation must
yield to the fact that you have no control over the resources that are
being used to represent it. a corollary is that storing any data in raw,
memory-like form externally (_including_ network protocols) is so stupid
and incompetent that programmers who do it without thinking should be
punished under law and required to prove that they have understood the
simplest concepts of computer design before they are ever let near a
computer again.
the third tenet of information representation is that data in _any_
external representation is _never_ outlived by the code, and that is an
inherent quality of external representation: the very reason you decided
to write it out to begin with is the reason it won't go away when the
code that wrote it did. this fact alone so fundamentally alters the
parameters of optimization of external representation from internal that
the only consequence of not heeding it is to wantonly destroy information.
now, there is one particular software company that profits by breaking
all possible understanding of information representation, and who makes
their very living from destroying the value of information previously
committed to the care of their software. this _started_ through sheer,
idiotic incompetence on their part, but turned into company policy only a
few years later: the company mission is now to destroy information like
no company has ever done before, for the sole purpose of causing the
customers to continue to pay for software that renders their past useless
and thus in need of re-creation, data conversion, etc.
| That is, if a thousand-element array of 4-bit bytes takes 512 bytes
| (according to time), and I write the contents of that array to disk, I
| expected to see a 512 byte file. Not so, apparently.
avoid the temptation to confuse internal with external representation,
and you will do fine. as soon as you think the two are even remotely
connected (as such -- going through a read/write process is precisely the
point showing how they are _not_ connected as such), you lose. big time,
and the way back to unconfusion is long and hard. witness all the C
victims who actually think it makes sense to dump memory to files. they
lose so badly you'd think somebody would learn from it, but no -- their
whole philosophy is to remain braindamaged in the face of danger, as that
kept them out of the scary, scary task of having to _think_ about things,
not something C programmers are very good at.
a 1000-element vector of (unsigned-byte 4) may optimally be stored as 500
bytes on a disk file if you are willing to _know_ what the file is like.
typically, hwoever, you would include metainformation that is not needed
once it has been processed and is in memory, versioning information, some
form of validation clues for the information (array bounds, etc), and in
all likelihood if you store binary data, some compression technique.
many arguments to the contrary notwithstanding, binary representation,
when it has been made to work, is _incredibly_ bloated compared to what
people set out believing it will be. making binary representation
efficient is so hard that most people give up, satisfied with gross
inefficiency. a Microsoft Word document is the typical example of how
unintelligently things _can_ be done when the real dunces are let at it.
you may actually be better off writing your vector of 4-bit bytes out as
hexadecimal digits, pretending that it is a huge number. Common Lisp
does not offer you any _huge_ help in hacking such notations back to more
common internal representations, but actually trying to time the work you
do during I/O has left many programmers bewildered by the cost of such a
simple operation. a disk read can easily take millions of times longer
than a memory read. whether you decode digits in that time or map bits
directly into memory is completely irrelevant to the system performance.
the conditions under which these low-level things matter are so different
from what normal people experience that they will have studied the topic
of internal and external representation very carefully before they need
to know the details.
| Allegro is my vendor.
well, no, Franz Inc is the vendor behind Allegro CL.
#:Erik
Erann Gat
> * Marco Antoniotti <mar...@parades.rm.cnr.it>
> | Is this a X3J13 proposal or a Franz proprietary one?
>
> how can _proposals_ be proprietary? sheesh.
Easily. Nothing prevents a vendor from issuing a proposal that includes
proprietary technology or trademarks. Politics often prevents such
proposals from being adopted. That's why they tend to be rare, and also
why it is entirely justified to ask the question.
Sheesh.
E.
Erik Naggum
| Nothing prevents a vendor from issuing a proposal that includes
| proprietary technology or trademarks. Politics often prevents such
| proposals from being adopted. That's why they tend to be rare, and also
| why it is entirely justified to ask the question.
well, if they tend to be rare, I'd have expected some _justification_ for
the implicit expectation that this was one of those rare cases.
however, to zoom out a bit, if people in the Common Lisp community wonder
why there's so (comparatively) little communal development activity, do
consider the responses received to the work that _is_ being done when it
is published. as far as I have seen, the response is generally "thanks.
now where's the rest?", no matter how much you do. it almost sounds like
a Norwegian tax collector, one of nature's strongest de-motivating forces.
#:Erik
Erann Gat
> * g...@jpl.nasa.gov (Erann Gat)
> | Nothing prevents a vendor from issuing a proposal that includes
> | proprietary technology or trademarks. Politics often prevents such
> | proposals from being adopted. That's why they tend to be rare, and also
> | why it is entirely justified to ask the question.
>
> well, if they tend to be rare, I'd have expected some _justification_ for
> the implicit expectation that this was one of those rare cases.
There was no implicit expectation that it *was* one of the rare cases,
just an implicit suspicion that it *might* be. Or maybe the original
poster didn't know or believe that proprietary proposals were rare.
Either way, IMO the question didn't warrant a disparaging response.
> however, to zoom out a bit, if people in the Common Lisp community wonder
> why there's so (comparatively) little communal development activity, do
> consider the responses received to the work that _is_ being done when it
> is published. as far as I have seen, the response is generally "thanks.
> now where's the rest?", no matter how much you do. it almost sounds like
> a Norwegian tax collector, one of nature's strongest de-motivating forces.
Rudeness can also be an inhibitor to communal development. All else being
equal most people will choose to work with someone who is friendly and
gives them the benefit of the doubt rather than someone who is rude and
condescending and makes worst-case assumptions about their motives.
E.
Duane Rettig
> Duane Rettig <du...@franz.com> writes:
>
> ...
> >
> > Because of this and further discussion, we have standardized on the
> > term "octet" when we describe 8-bits specifically. The proposal that
> > resulted from this is part of a design of an I/O system, which will
> > complement the current Gray streams system we currently have, but
> > which will address various issues of stream mutivalency, multibyte
> > character sets and locales, speed, and a few other problems that the
> > CL spec and Gray streams don't currently address. We hope to publish
> > this proposal soon, long before our next major release.
> >
>
> Is this a X3J13 proposal or a Franz proprietary one?
It is not an X3J13 proposal. I am not sure what you mean by
a Franz proprietary one.
Our intention in putting this proposal out is to solicit
technical reactions to the merits and flaws of the design,
and to solicit suggestions for improvement. As a document,
it will have a copyright, but as an api spec - well, I don't
think it's possible to copyright an api. The ideas, once
broadcast, will become public and thus anyone could submit a
modified proposal as their own, if they wanted to.
As for X3J13, the proposal may come to be appropriate for submission
eventually, but we're certainly not there yet and are taking this
one step at a time.
Erik Naggum
| Rudeness can also be an inhibitor to communal development. All else being
| equal most people will choose to work with someone who is friendly and
| gives them the benefit of the doubt rather than someone who is rude and
| condescending and makes worst-case assumptions about their motives.
I'm happy that you, too, consider Marco Antoniotti's response _rude_
towards Franz Inc, and that you support my point: if you suspect the
worst of motives when you receive a gift from somebody, other people
generally don't give you much _more_ for free from then on.
as for rudeness in this particular case, maybe you just need to learn to
live with the fact that people find _you_ rude and _disgusting_ to deal
with when you don your halo and fight to save the world from a rudeness
you find only when _you_ make worst-case assumptions about the motives of
others, but I guess that's OK since you do it, wearing that halo. I wish
you had learned something, but you really learned _nothing_ from the last
time we had this discussion. *sigh*
and if you are such a champion of benefit of the doubt, consider making
some actions of yours conform to your self-serving claims some day. it
would be a little less disgusting to behold a reeking hypocrite actually
follow his _own_ advice once in a while, at least.
#:Erik
Marco Antoniotti
Duane Rettig <du...@franz.com> writes:
> Marco Antoniotti <mar...@parades.rm.cnr.it> writes:
> >
> > Is this a X3J13 proposal or a Franz proprietary one?
>
> It is not an X3J13 proposal. I am not sure what you mean by
> a Franz proprietary one.
You answer below.
> Our intention in putting this proposal out is to solicit
> technical reactions to the merits and flaws of the design,
> and to solicit suggestions for improvement. As a document,
> it will have a copyright, but as an api spec - well, I don't
> think it's possible to copyright an api. The ideas, once
> broadcast, will become public and thus anyone could submit a
> modified proposal as their own, if they wanted to.
>
> As for X3J13, the proposal may come to be appropriate for submission
> eventually, but we're certainly not there yet and are taking this
> one step at a time.
That is all I wanted to know and I say it is fair enough.
As usual the problem is "what happens if Harlequin decides that Franz
proposal is not-what-they-wanted and vice-versa". Then we may end up
with two different implementations for I/O stuff and the whole Common Lisp
community suffers.
Marco Antoniotti
Erik Naggum <er...@naggum.no> writes:
> * Marco Antoniotti <mar...@parades.rm.cnr.it>
> | Is this a X3J13 proposal or a Franz proprietary one?
>
> how can _proposals_ be proprietary? sheesh.
Duane said that the "proposal" is not an X3J13 one (yet). The
question I had in mind was: can Harlequin implement the API described
in a Franz Copyrighted document? Duane implies a "yes" in his answer.
This is what I read here. Apologies if my question was too concise.
> work on this topic was "commissioned" at the last X3J13 meeting, and
> Franz accepted the responsibility to oversee the work if not write it.
That is good. No questions about it.
Fernando D. Mato Mira
> Ouch! Next week I'm planning to write code that generates binary files
> to a certain standard (Midi music data). I haven't checked the docs
http://www-ccrma.stanford.edu/CCRMA/Software/cm/examples/qp.lisp
--
((( DANGER )) LISP BIGOT (( DANGER )) LISP BIGOT (( DANGER )))
Fernando D. Mato Mira
Real-Time SW Eng & Networking
Advanced Systems Engineering Division
CSEM
Jaquet-Droz 1 email: matomira AT acm DOT org
CH-2007 Neuchatel tel: +41 (32) 720-5157
Switzerland FAX: +41 (32) 720-5720
www.csem.ch www.vrai.com ligwww.epfl.ch/matomira.html
Andrew Cooke
"Fernando D. Mato Mira" <mato...@iname.com> wrote:
> Andrew Cooke wrote:
>
> > Ouch! Next week I'm planning to write code that generates binary
files
> > to a certain standard (Midi music data). I haven't checked the docs
>
> http://www-ccrma.stanford.edu/CCRMA/Software/cm/examples/qp.lisp
unfortunately, that (and a few variants I tried) doesn't work
(truncated versions look like they do exist, but I don't have permission
to read them). could you repost?
also, thanks for all the replies.
andrew
Tim Bradshaw
> As usual the problem is "what happens if Harlequin decides that Franz
> proposal is not-what-they-wanted and vice-versa". Then we may end up
> with two different implementations for I/O stuff and the whole Common Lisp
> community suffers.
I don't think this follows. Two different implementations of
differing (publically documented) APIs gives people the chance to
explore the characteristics of either, and then to arrive at some
later version, which may either be a derivative of one or the other, a
unified version or something different again, and which could then be
standardised. There may be some temporary pain while divergent
versions exist, but this is better than prematurely standardising
something which then hangs around your neck for ever.
--tim
Duane Rettig
> Duane Rettig <du...@franz.com> writes:
> > As for X3J13, the proposal may come to be appropriate for submission
> > eventually, but we're certainly not there yet and are taking this
> > one step at a time.
>
> That is all I wanted to know and I say it is fair enough.
>
> As usual the problem is "what happens if Harlequin decides that Franz
> proposal is not-what-they-wanted and vice-versa". Then we may end up
> with two different implementations for I/O stuff and the whole Common Lisp
> community suffers.
I disagree that the community would suffer. Initiating dialog always
carries with it the risk of disagreement, even to the point of impasse
in the extreme case. But without dialog, change cannot occur.
We have made good use of the Gray streams proposal for the last 10
years. Others have started using it more recently, and it has served
us all well. But there are several problems with the proposal, and
we have a good idea as to how to do I/O better in several ways. This
new proposal that we will make available encapsulates those ideas.
If the ideas that this proposal espouses are completely wrong, then
it will become clear in the process of being made public. But if
these ideas catch on, then they will be implemented, whether portably
or not, and the whole Lisp community will win, not suffer.
One other aspect of this: Franz has the choice of how to disseminate
this proposal. We could just play the cards close and show it first
at our next release. Or, we could show it sooner and allow both the
community and ourselves to benefit from the ensuing dialog. We have
chosen the latter approach.
Craig Brozefsky
> Erik Naggum <er...@naggum.no> writes:
>
> > * Marco Antoniotti <mar...@parades.rm.cnr.it>
> > | Is this a X3J13 proposal or a Franz proprietary one?
> >
> > how can _proposals_ be proprietary? sheesh.
>
> Duane said that the "proposal" is not an X3J13 one (yet). The
> question I had in mind was: can Harlequin implement the API described
> in a Franz Copyrighted document? Duane implies a "yes" in his answer.
> This is what I read here. Apologies if my question was too concise.
Yah, they can implement it. In the other direction, I am implementing
something described in Harlequin copyrighted documentation for
CMUCL(should be largely portable tho), the CommonSQL package.
Unlike Erik, I will not spank you now, even if that is what you
want.
--
Craig Brozefsky <cr...@red-bean.com>
Free Scheme/Lisp Software http://www.red-bean.com/~craig
"Hiding like thieves in the night from life, illusions of
oasis making you look twice. -- Mos Def and Talib Kweli
Erik Naggum
| The question I had in mind was: can Harlequin implement the API described
| in a Franz Copyrighted document?
this "question" is an insult to people who provide information to the
community. just because they see no reason to give away everything and
every right to their work does not mean that they are bastards out to
control people's reasonable courses of action. copyright is a fairly
well-defined legal concept -- it does not preclude understanding and
using the information and ideas whose expression is protected from
reproduction, and never has precluded such use. in particular, copyright
does not affect the implementability of a specification.
| This is what I read here. Apologies if my question was too concise.
I really think you need to study some copyright law. unfounded fears
such as the ones you have expressed can lead to no good, especially if
they go unchecked and you actually believe that affixing a copyright
notice to a specification has any bearing on its implementability.
sometimes, I wonder if the reason people favor free software is that they
have zero clue about the freedoms and rights they actually _have_ within
the boundaries of intellectual property law, and maybe the bone-headed
intellectual-property lawyers of, say, DVD-CCA, tend to reinforce their
fears, too, but hysterics like fearing that a copyright notice would not
allow others to implement the functions described in a document, _could_
explain why people who want source code to everything consistently fail
to grasp the meaning of "license", in particular, that the GPL is in fact
a _license_ that grants them a bunch of rights they otherwise do not
have, and that if they are not granted all these rights by the only legal
entity that can grant them, they are in fact violating intellectual
property laws, whether they get prosecuted for it or not, or, indeed,
whether they are cast as world-class heroes for it or not. arguing such
topics requires people to know the rights they have _under_ the law --
intellectual property is one of those areas where the law grants rights
that cannot be abridged by contract or agreement. few "freedom fighters"
realize that they are often storming open doors and fighting for what
they already have. some "freedom fighters" implicitly accuse others of
denying them rights they cannot be denied, except to the degree that they
willfully reject the option of exercising them, for which hardly anyone
else can be held responsible.
a population which has natural rights and has also been granted rights
under the law, but which is so ignorant of their rights that they believe
anyone can take them away from them at will, will live in such fear of
the law itself that both rights and law lose their meaning. this is what
is happening to the ignorant hacker community, quite unlike what _could_
happen to the knowledgeable hackers of only a few years past. "ignorance
of the law is no excuse" cuts both ways. incompetence in the search for
freedom will only cause them to find the freedom to be incompetent.
#:Erik
Marco Antoniotti
Duane Rettig <du...@franz.com> writes:
> Marco Antoniotti <mar...@parades.rm.cnr.it> writes:
>
> > Duane Rettig <du...@franz.com> writes:
>
> > > As for X3J13, the proposal may come to be appropriate for submission
> > > eventually, but we're certainly not there yet and are taking this
> > > one step at a time.
> >
> > That is all I wanted to know and I say it is fair enough.
> >
> > As usual the problem is "what happens if Harlequin decides that Franz
> > proposal is not-what-they-wanted and vice-versa". Then we may end up
> > with two different implementations for I/O stuff and the whole Common Lisp
> > community suffers.
>
> I disagree that the community would suffer. Initiating dialog always
> carries with it the risk of disagreement, even to the point of impasse
> in the extreme case. But without dialog, change cannot occur.
I understand this, and agree.
> We have made good use of the Gray streams proposal for the last 10
> years. Others have started using it more recently, and it has served
> us all well. But there are several problems with the proposal, and
> we have a good idea as to how to do I/O better in several ways. This
> new proposal that we will make available encapsulates those ideas.
This is a good thing.
> If the ideas that this proposal espouses are completely wrong, then
> it will become clear in the process of being made public. But if
> these ideas catch on, then they will be implemented, whether portably
> or not, and the whole Lisp community will win, not suffer.
I also agree on this. I am just wary of the "potably or not" in your
sentence.
> One other aspect of this: Franz has the choice of how to disseminate
> this proposal. We could just play the cards close and show it first
> at our next release. Or, we could show it sooner and allow both the
> community and ourselves to benefit from the ensuing dialog. We have
> chosen the latter approach.
And you should be commended for that. You made a good choice.
Just to clarify. I have been pleased by the main vendors (Franz and
Harlequin) in the past. There are still issue I disagree with,
but all in all I am grateful for what the vendors have been doing.
If once in a while I sound too critical, it's because I stepped out of
bed with the wrong foot that day :)
Fernando D. Mato Mira
> In article <38903844...@iname.com>,
> "Fernando D. Mato Mira" <mato...@iname.com> wrote:
> > Andrew Cooke wrote:
> >
> > > Ouch! Next week I'm planning to write code that generates binary
> files
> > > to a certain standard (Midi music data). I haven't checked the docs
> >
> > http://www-ccrma.stanford.edu/CCRMA/Software/cm/examples/qp.lisp
>
> unfortunately, that (and a few variants I tried) doesn't work
> (truncated versions look like they do exist, but I don't have permission
> to read them). could you repost?
Weird. You shouldn't have prolems with either the complete URL or the
top-level page
Anyway, Common Music home page:
http://www-ccrma.stanford.edu/CCRMA/Software/cm/cm.html
You can get to the above example of how to play MIDI files from there.
Isn't it wonderful how now you'll be able to help adding Level 1 write
support instead of doing everything from scratch?
Regards,
Fernando D. Mato Mira
> reproduction, and never has precluded such use. in particular, copyright
> does not affect the implementability of a specification.
Itchy question: What happens when the specification is (partly) expressed in a
copyrighted
C header file? Even if you type yourself every byte, it'll be essentially the
same thing.
Tim Bradshaw
> I also agree on this. I am just wary of the "potably or not" in your
> sentence.
I assumed that Duane meant there might be non-portable implementations
of the same API -- rather like native CLOS vs PCL.
--tim
Robert Monfera
Craig Brozefsky wrote:
> Yah, they can implement it. In the other direction, I am implementing
> something described in Harlequin copyrighted documentation for
> CMUCL(should be largely portable tho), the CommonSQL package.
How does it relate to Paul Meurer's SQL-ODBC package?
Robert
Craig Brozefsky
It doesn't actually. It is built on top of Pierre Mai's MaiSQL, which
provides the database interfaces. I implemented a [...] sql syntax
like CommonSQL. The SQL-ODBC [...] syntax is actually different than
the CommonSQL one. SQL-ODBC also did not have an OO interfce when I
last looked at it. I am implemented that presently, and have
everything except for :join slots done. I am making some improvements
on CommonSQL as well, which are all compatible with the published
interface.
Erik Naggum
| Itchy question: What happens when the specification is (partly) expressed
| in a copyrighted C header file? Even if you type yourself every byte,
| it'll be essentially the same thing.
invoke "fair use" and claim that you cannot copyright numerical constants.
copyright does not cover reproducing functional equivalence, so you could
argue that you're not reproducing more than you have to in order to use
the specification for its intended purpose (which by law is prevented
from being "to screw the readers" :).
#:Erik
Marco Antoniotti
Erik Naggum <er...@naggum.no> writes:
> * Marco Antoniotti <mar...@parades.rm.cnr.it>
> | The question I had in mind was: can Harlequin implement the API described
> | in a Franz Copyrighted document?
>
> this "question" is an insult to people who provide information to the
> community.
My original question and also what I said afterward were not meant to
be an insult to anybody. If they have been read in this way I
apologize.
However, the result of my question was to read many postings that
clarified many things about Franz standing, and about their upcoming
proposal for I/O and other things. I believe this is a good thing.
> just because they see no reason to give away everything and
> every right to their work does not mean that they are bastards out to
> control people's reasonable courses of action.
I never implied this. And, as I said, there are issues I disagree
with with the vendors' policies. If this is offensive, there is
nothing I can do about it. In this specific case, Duane Rettig's
response was definitively not something I could disagree with.
> copyright is a fairly
> well-defined legal concept -- it does not preclude understanding and
> using the information and ideas whose expression is protected from
> reproduction, and never has precluded such use. in particular, copyright
> does not affect the implementability of a specification.
Fine. But in this specific case my problem is also that Harlequin (or
another vendor, or the CMUCL and CLisp crowd) will not be "bound" (for
an imprecise definition and use of the term) to implement anything.
Should the (amended through public discussion) proposal be accepted by
X3J13, Harlequin and the others, would have more or less of a "duty"
(I am once again imprecise) to implement the proposal.
Now the question for you is: is this a "legitimate" issue to raise?
> | This is what I read here. Apologies if my question was too concise.
>
> I really think you need to study some copyright law.
Yes and no. I can just ask you :)
I cut the rest. It was interesting to read and I learned something.
Anyway, let me assure you that I know I may be naive in many ways, but
in this case I definitively do not feel like I did anything
particularly wrong.
Tim Bradshaw
> Fine. But in this specific case my problem is also that Harlequin (or
> another vendor, or the CMUCL and CLisp crowd) will not be "bound" (for
> an imprecise definition and use of the term) to implement anything.
> Should the (amended through public discussion) proposal be accepted by
> X3J13, Harlequin and the others, would have more or less of a "duty"
> (I am once again imprecise) to implement the proposal.
But the point is that until there exist a number of implementations of
some better proposal, or proposal*s*, J13 would be seriously
ill-advised to try and sit down and try and impose some standard on
everyone, because it might just not be implementable by anyone. I am
not confident of the ability of J13 (or *any* organisation) to sit
down and arrive at some reasonable standard in an area where there is
no experience of implementation. The result of that would be a
disaster in my opinion (and I'm pretty sure that this is the opinion
of most J13 members too, thank goodness!).
So I am really in favour of vendors experimenting with extensions.
--tim
Bruce L. Lambert
Thanks for the informative treatise on internal and external
representations. With any luck, it shold be a while until I make this
mistake again.
-bruce
Marco Antoniotti
Tim Bradshaw <t...@cley.com> writes:
I agree with this as well.
I guess I will not intervene anymore on these subjects. It would take
a very long post with a lot of premises, just to ask a question. :)
Kaelin Colclasure
news:44sbz3...@beta.franz.com...
[...]
> If the ideas that this proposal espouses are completely wrong, then
> it will become clear in the process of being made public. But if
> these ideas catch on, then they will be implemented, whether portably
> or not, and the whole Lisp community will win, not suffer.
Bravo! Well said.
> One other aspect of this: Franz has the choice of how to disseminate
> this proposal. We could just play the cards close and show it first
> at our next release. Or, we could show it sooner and allow both the
> community and ourselves to benefit from the ensuing dialog. We have
> chosen the latter approach.
And as a paying customer of Franz, I'm very pleased with this approach.
The last thing the CL community needs is yet another tired replay of
the kind of paranoid vendor-bashing that killed News and stuck us
with X-Windows, etc.
-- Kaelin
Erik Naggum
| Now the question for you is: is this a "legitimate" issue to raise?
implementing a standard is always voluntary. this may annoy some people
(it annoys me _greatly_ at times, like when the stupid Norwegian Language
Council incredulously insists on writing "13.2.00" for today, for the
single, simple, stupid reason that the Swedes write "2000-02-13", and we
can't do anything the Swedes do unless it's really stupid and then not
until they figured out that it was stupid and did something else that we
obviously can't do at that point, but I digress irresponsibly). even
when some proposal is put into a standard, the user community has to be
responsible and take issue with half-witted or bogus implementations.
| > I really think you need to study some copyright law.
|
| Yes and no. I can just ask you :)
yikes! like most other people who give legal advice on the Net, what I
have said on legal matters have turned out to be wrong at some later
date, so this is an almost _frightening_ level of "trust". like the
other day, British authorities came up with this bill that would make it
illegal _not_ to decode files or messages stored on your computer, as if
the bloody obvious assault on every citizen's assumption of innocence
until proven guilty would _not_ be to store files or messages on their
computer to which they did not in fact _have_ a decoding key. the police
often plant evidence in drug cases to incarcerate people they just don't
like, and now they can do the same with computer people they just don't
like. however, a few weeks ago, I would have said this was impossible to
implement under the European Human Rights Convention, but what do you
know? I was wrong: in the interest of removing personal freedom and
security from people's lives when they do something politicians are
afraid of out of sheer, mind-numbing ignorance, _anything_ goes, and
_anywhere_ in the world, even England, with their Magna Carta heritage.
#:Erik
Duane Rettig
Call for Comments: The New Allegro CL Streams Design
Location:
http://www.franz.com/support/docs/simple-stream.htm
Please respond to bu...@franz.com, with the subject line:
Comments on streams design
We at Franz, Inc. are pleased to make our new simple-streams
design available to the public for review and comment. Whether you
are a customer or a competitor, we are interested in what you
think of this document, good or bad. Comments may be made
toward any aspect of the design that you wish, including its
applicability to general CL implementation. With that in mind,
please note that the document itself has a large amount of
Allegro CL specific wording, and is thus probably not appropriate
for non-Franz Common Lisps. However, the intention of the
document is to promulgate implementation-independent concepts,
and the basic design should be portable.
The design is known to work in the Allegro CL implementation,
in a version that will become 6.0 but which is not quite ready
for beta testing.
We may answer mail that comes in, depending on its nature,
but please don't count on us answering all such mail. In
particular, we will _not_ answer the following questions
beyond the comments given here:
Questions we will not comment on:
- When will 6.0 be released? The general answer is "Sometime this
year." However, the technical staff will not give a more
specific answer than that.
- How fast is it? Other than preliminary timings and
optimizations, which indicate that the design meets our own
performance goals, we have not done thorough optimization on
the implementation, and thus do not yet have any numbers to
offer you. We may have such numbers in the future.
Enjoy.