Lisp-NYC has provided to Google positive evaluations for seven of the
nine Summer of Code projects Google sponsored through Lisp NYC. One
project retired in the face of unexpected difficulties, and another --
cl-sockets -- was given a negative evaluation. The student has now done
additional work (after the deadline) and is hoping for a favorable
re-evaluation.
The mentors (moi included) do not see enough to justify spending more of
our time on this, but if anyone wants to eyeball the latest cl-sockets
and sing its praises I for one will reconsider.
Here it is: http://common-lisp.net/project/cl-sockets/
I will see your feedback if you post it here, or you can send me an
email, in confidence or otherwise.
The remainder of this post is the original proposal (or at least a late
revision -- I might have missed the last actually submitted to Google).
kenny
Project Title
Lisp Sockets
Synopsis
One of the Great Complaints about Lisp is it's lack of standard
sockets. As Lisp was born before the Internet became a big deal sockets
did not make it into the CL standard. The current status of Lisp sockets
is every Lisp has it's own implementation and various "universal" socket
libraries go through each implementation's socket library. The Lisp
Sockets project exists to make open source projects more shareable, by
eliminating an implementation dependency by erasing this black mark from
the history of Lisp. The proposal is to create single socket library
with a standard interface for all Lisp implementations in all
environments. This will be accomplished by using a portable OS FFI
(Foreign Function Interface) to gain access to native C socket
libraries. The particular FFI used will be Hello-C. [1] Hello-C is a
Lisp project which allows Lisp code to include C headers and link
seamlessly with C libraries. Here it will be used to write an interface
directly to BSD sockets on Unix, Winsock on win32, and either BSD
sockets or native OSX sockets on OSX.
Deliverables
At the completion of this project Lisp will have a truly universal
socket library with an interface similar to that of BSD sockets. This is
critical to Lisp so that programmers do not have to worry about system
specific features. Programmers will be able to write network driven
applications in Lisp easily and deploy them across all platforms without
modification.
Project Details
This project will involve defining a standard socket interface,
independent of operating system and Lisp implementation. This interface
will in turn integrate with the various system's socket interface with
the help of Hello-C. Interfacing with the various API's will also
require this project to transparently handle any/all implementation
specific idiosyncrasies in order to provide a truly universal socket
library. Because of differences between various current socket
interfaces (and theoretical future socket implementations), Lisp sockets
will offer a lowest common denominator socket interface for true system
transparency. Lisp Sockets will also offer an interface to possible
platform specific features/bugs for easier system specific integration
with legacy code.
Project Schedule
In many ways this project has already begun. I have gotten the go ahead
for this project from NYCLisp [2] mentor Kenny Tilton [3] and we have
begun to flesh out the details. The following areas roughly corresponds
to two week periods of work that will be completed in order to achieve
truly system transparent sockets.
- API
- Regression Test
- CLisp + NetBSD
- Expansion and testing to other Unix-like systems
- AllegroCL + win32
Bio
I am a senior at Polytechnic University [4] majoring in Computer
Science [5] and a graduate of Stuyvesant High School. [6] I am the
president of Poly's chapter of the ACM. [7] I am also one of the
founders and a regular programmer at Poly's programming team meetings
[8] where we solve ACM contest type problems weekly. Algorithms is
generally the field I am most interested in. I have worked on many
socket related project in various languages. [9] Last summer and through
the year I was working on an NSF funded project, ForNet [10], for the
majority of my time at Poly. ForNet is a distributed forensics network.
It provides a scalable network logging mechanism to aid forensics over
wide area networks. Fornet had me working on many network and
algorithmic issues. Issues addressed by myself for the project included:
- Fast packet scheduling within a widely deployed forensics network
- Long term data storage solutions for huge amounts of forensics related
network data (think gigabits per second)
- Fast decoding of network protocols for packet analysis of relevant
data including TCP/IP, DNS, HTTP, etc
- 3D Visualization of a massive amount of network related events in
real-time across large networks
- Research into algorithms of packet analysis for significant payloads
(polymorphic code, hyper-slow portscans, sensitive data distribution, spam)
- Privacy issues
Numerous recommendations available from Polytechnic University upon request.
[1] http://www.alphageeksinc.com/cgi-bin/lispnyc.cgi?HelloC
[2] http://www.lispnyc.org/
[3] http://tilton-technology.com/
[4] http://www.poly.edu/
[5] http://cis.poly.edu/
[6] http://www.stuy.edu/
[7] http://acm.poly.edu/
[8] http://cis.poly.edu/jsterling/PolyProgrammingTeam/
[9]
http://www.zoo-crew.org/gallery/view_photo.php?set_albumName=halloween2004&id=PA300014
[10] http://isis.poly.edu/projects/fornet/
> One project retired in the face of unexpected difficulties,
Which one was that?
Matt
--
"You do not really understand something unless you can
explain it to your grandmother." — Albert Einstein.
So, this is new?
> Lisp-NYC has provided to Google positive evaluations for seven of the
> nine Summer of Code projects Google sponsored through Lisp NYC. One
> project retired in the face of unexpected difficulties, and another --
> cl-sockets -- was given a negative evaluation. The student has now done
> additional work (after the deadline) and is hoping for a favorable
> re-evaluation.
>
> The mentors (moi included) do not see enough to justify spending more of
> our time on this, but if anyone wants to eyeball the latest cl-sockets
> and sing its praises I for one will reconsider.
Does April Fools come twice this year?
After not getting much done for a couple of months, suddenly Mr Wizard
will crank it out in days?
Frankly, that claim simply increases my negative opinion.
--
In Common Lisp quite a few aspects of pathname semantics are left to
the implementation.
> > One project retired in the face of unexpected difficulties,
>
> Which one was that?
>
> Matt
The retired project is PLT-DB. That student announced before the Sept 1
deadline that there was not enough time remaining, and that he was
giving up.
My SoC project, Erlisp, should probably also be added to the list of
incomplete projects. My process-linking code, the most important piece
of my SoC work, is still not working. So this brings it to 6 out of 9?
Still not a bad result for the first try.
Eric
GP lisper wrote:
> On Wed, 21 Sep 2005 15:57:08 GMT, <kti...@nyc.rr.com> wrote:
>
>>We have a drama unfolding.
>
>
> So, this is new?
As far as Lisp NYC's experience with the Google Summer of Code, yes. I
wonder how many other brush fires have broken out amongst the Google 400.
>
>
>>Lisp-NYC has provided to Google positive evaluations for seven of the
>>nine Summer of Code projects Google sponsored through Lisp NYC. One
>>project retired in the face of unexpected difficulties, and another --
>>cl-sockets -- was given a negative evaluation. The student has now done
>>additional work (after the deadline) and is hoping for a favorable
>>re-evaluation.
>>
>>The mentors (moi included) do not see enough to justify spending more of
>>our time on this, but if anyone wants to eyeball the latest cl-sockets
>>and sing its praises I for one will reconsider.
>
>
> Does April Fools come twice this year?
>
> After not getting much done for a couple of months, suddenly Mr Wizard
> will crank it out in days?
>
> Frankly, that claim simply increases my negative opinion.
Agreed, and the student has received copious similar feedback (thx for
reinforcing). But if he pulls it off, it will make for a great war
story. So I am still judging the code on its merits.
Note that I also happened to mentor another "universally portable Lisp
doo-dad" project, CFFI ala Luis, and have that excellent example against
which to compare. Luis created a regression test and actually ran it
against every Lisp he could get his hands on, including going to
proprietary fringe Lisps and requesting evaluation copies for this purpose.
the idea is that theoretically portable is nice, but then the work
begins of finding out how this Lisp or that manages to break the
regression test, possibly via some bug which can be reported and then
cured by implementors. At the end of this exercise one can claim to have
delivered a portable library. The cl-sockets student is stopping at one
Lisp on each platform. And has been told that that will not suffice.
Sorry, the above is just a very long way of saying that it is not just
the code that is being judged, but also the verification of "in fact"
portability. Quality of documentation is another point of concern, tho
my many fans know I give that less weight than the code itself and
abundant examples reaching into the edge cases of socketry.
--
Kenny
Why Lisp? http://wiki.alu.org/RtL_Highlight_Film
"I've wrestled with reality for 35 years, Doctor, and I'm happy to state
I finally won out over it."
Elwood P. Dowd, "Harvey", 1950
Yes, CFFI is very nice. Has anyone tried replacing UFFI in CLSQL with
CFFI, or something similar? The mailing list is full of technical
details, but I missed any practical applications (outside of Kenny
code).
GP lisper wrote:
> On Wed, 21 Sep 2005 18:56:09 GMT, <kti...@nyc.rr.com> wrote:
>
>>Note that I also happened to mentor another "universally portable Lisp
>>doo-dad" project, CFFI ala Luis, and have that excellent example against
>>which to compare. Luis created a regression test and actually ran it
>>against every Lisp he could get his hands on, including going to
>>proprietary fringe Lisps and requesting evaluation copies for this purpose.
>
>
> Yes, CFFI is very nice. Has anyone tried replacing UFFI in CLSQL with
> CFFI, or something similar? The mailing list is full of technical
> details, but I missed any practical applications (outside of Kenny
> code).
>
Practical applications of CLSQL? There must be at least one... oh! You
mean of CFFI? You are asking what is the practical application of a
portable FFI? Or do you think UFFI is either portable or efficient on
all platforms?
(<g> My serious answer to your question is in there somewhere. <\g>)
Well, I dunno. CLSQL looks rather poor so far in comparison to the
PerlAPI I used for years. A very big difference my start-up time (and
when I learned the perlAPI, I didn't even know SQL, but had a data
miner running in 45 minutes), and problems connecting to existing
tables on linux. Some sort of reader hack with [], maybe for
backwards compatibility reasons, which makes the syntax difficult to
read and is the source of many errors from parens trained fingers.
CLSQL probably works fine in brand-new setups that use the view-class
exclusively.
> mean of CFFI? You are asking what is the practical application of a
> portable FFI? Or do you think UFFI is either portable or efficient on
> all platforms?
I just want to follow some CFFI breadcrumbs....in something I already
understand preferably.
> (<g> My serious answer to your question is in there somewhere. <\g>)
Dang, I looked under all the rocks too and still didn't find it!
GP lisper wrote:
Dude, CFFI is more extensive and portable than UFFI. With CFFI CLSQL
could be more portable (well, if it just uses UFFI, I do not actually
know this, but that would be the idea). Practical enough?
Hmmmm... maybe you are thinking "what good does this do GPLisper? CLSQL
already runs on my platform." If so, riiiiiiggghhhht, this is about CL
users on the whole having more access to libraries because on the whole
CFFI bindings are more portable. As for any advantage over UFFI, uh, I
am the wrong person to ask, but IIUC CFFI is more efficient than UFFI
under certain implementations. Apologies now if that is gross
misinformation, but to me the only thing that matters is making all
C/C++ libraries effectively also Lisp libraries (with Verrazano writing
the bindings for us).
hth
> Well, I dunno. CLSQL looks rather poor so far in comparison to the
> PerlAPI I used for years. A very big difference my start-up time
> (and when I learned the perlAPI, I didn't even know SQL, but had a
> data miner running in 45 minutes), and problems connecting to
> existing tables on linux. Some sort of reader hack with [], maybe
> for backwards compatibility reasons, which makes the syntax
> difficult to read and is the source of many errors from parens
> trained fingers. CLSQL probably works fine in brand-new setups that
> use the view-class exclusively.
You don't have to use the CLSQL reader syntax and you don't have to
use view classes. If you leave both of them alone you have plain,
string-based SQL like in Perl. Where's the problem?
CL-USER 1 > (asdf:oos 'asdf:load-op :clsql)
; [snip]
NIL
CL-USER 2 > (use-package :clsql)
T
CL-USER 3 > (connect '("orcl" "edi" "xxx") :database-type :oracle)
; [snip]
#<CLSQL-ORACLE:ORACLE-DATABASE orcl/edi OPEN 206A027C>
CL-USER 4 > (query "select sysdate from dual")
(("2005-09-22 10:21:06+00"))
("SYSDATE")
CL-USER 5 > (execute-command "create table foo (fnork number)")
CL-USER 6 > (execute-command "insert into foo values (42)")
CL-USER 7 > (query "select fnork from foo")
((42))
("FNORK")
CL-USER 8 > (query "select fnork from foo" :flatp t)
(42)
("FNORK")
Granted, there are some things like bindings for Oracle that are in
DBI and not in CLSQL but I doubt you used them in your
45-minutes-without-knowing-SQL solution. Hey, it's open source, add
what you like...
Cheers,
Edi.
--
Lisp is not dead, it just smells funny.
Real email: (replace (subseq "spam...@agharta.de" 5) "edi")
> The mentors (moi included) do not see enough to justify spending
> more of our time on this, but if anyone wants to eyeball the latest
> cl-sockets and sing its praises I for one will reconsider.
First of all thank you for sponsoring, and to the student for working
on, this common networking API for Common Lisp. API design is hard,
especially for generic APIs.
> Here it is: http://common-lisp.net/project/cl-sockets/
> I will see your feedback if you post it here, or you can send me an
> email, in confidence or otherwise.
Quick feedback on what I saw:
* The documentation is sparse (and hard to read in my browser because
of weird formatting). OK, so everything is supposed to be like BSD
Sockets.
* The API style is decidedly un-Lispy - all functions return lists,
the first element of which is an error indication (T for error, NIL
for success). Code using this will look ugly. Please consider
using multiple values and/or conditions for error indication
instead.
* The API encourages the use of AF-dependent interfaces such as
gethostbyaddr() (or (gethostbyaddr)), which makes it hard to
accomodate IPv6.
* This relates to the representation of (IPv4) addresses, which in the
cl-socket API is really weird, namely a list of four integers or one
of several symbols such as CL-SOCKET:INADDR_ANY. It would be
beneficial to use a distinct address type here, so that other
address families can be supported. I don't care about the
representation of these types, as long as there are convenience
functions to convert between those objects and literals and the
lookup functions support them too.
They way it is specified, this API doesn't provide much value-add
compared to calling the C socket library functions through FFI, except
for some conversion from native Lisp data (such as the
quad-integer-list representation of IPv4 addresses) to C structs.
Of the sockets functions that I consider essential, CL-SOCKETS is
missing at least two, namely sendto() and recvfrom().
It also doesn't say how I can write to/read from stream (TCP) sockets.
I assume that in "real" Lisp networking APIs, stream sockets can be
used like normal streams. If so, you not only have to specify and
implement that accordingly, but also have to think about issues such
as buffering and format conversion. If not, you need operations to
read and write to stream sockets. The descriptions of CL-SEND and
CL-RECV indicate that they are for datagram sockets, although in
principle they could be extended to stream sockets (but remember TCP
streams don't have message boundaries).
All data ("msg") arguments are specified as Lisp strings, but TCP/IP
operates in terms of octets. How are string contents converted to and
from octets? It would be much easier to define semantics for binary
(ARRAY (*) (UNSIGNED-BYTE 8)) messages - which are also more natural
to use for some types of applications - and then maybe specify
implicit conversion rules between those binary arrays and other
arrays, such as strings. Also, it would be convenient to use :START
and :END arguments to specify areas of interest of those arrays, and
to get rid of the "len" arguments, since Lisp arrays carry their
lengths with them.
> The remainder of this post is the original proposal (or at least a
> late revision -- I might have missed the last actually submitted to
> Google).
[...]
> Deliverables
> At the completion of this project Lisp will have a truly
> universal socket library with an interface similar to that of BSD
> sockets.
Personally I think similarity to the BSD Sockets API is not such a
good goal anyway. The java.net API would be a better starting point.
Regards,
--
Simon.
Yes, I think I was looking into SELECT too much, it doesn't have AS,
so my sorting won't work anyway. QUERY somewhat solves that problem,
but it looks like I need both SELECT and QUERY. The [] was annoying
because in slime or lisp-mode, you don't get paren matching for []
without a cliini fix:
http://paste.lisp.org/display/11887
The OS lettercase problems were solved in my.cnf, with some magic from
'mmv' needed. I've wrapped enough of the reader-syntax in functions
to generally ignore it for now. </rant>
Oh, you wanted a sample problem:
select winner AS 'team',count(winner) AS 'wins' from games where
((home="LAA" AND visitor="BOS") OR (home="BOS" AND visitor="LAA"))
group by winner order by wins desc;
where the team names "LAA" and "BOS" are parameters in a function
call.
+------+------+
| team | wins |
+------+------+
| BOS | 6 |
| LAA | 4 |
+------+------+
mysql> select * from games limit 4;
+----------+--------+-------+---------+------+--------+--------+-------+--------+--------+-----+--------+
| date | gnumbr | start | visitor | home | v_runs | h_runs | total | margin | winner | hrv | gameid |
+----------+--------+-------+---------+------+--------+--------+-------+--------+--------+-----+--------+
| 20050403 | 1 | 1205 | BOS | NYY | 2 | 9 | 11 | 7 | NYY | H | 1 |
| 20050404 | 1 | 785 | KC | DET | 2 | 11 | 13 | 9 | DET | H | 2 |
| 20050404 | 1 | 905 | CLE | CWS | 0 | 1 | 1 | 1 | CWS | H | 3 |
| 20050404 | 1 | 905 | OAK | BAL | 0 | 4 | 4 | 4 | BAL | H | 4 |
+----------+--------+-------+---------+------+--------+--------+-------+--------+--------+-----+--------+
$sth = $dbh->prepare( 'select winner AS 'team',count(winner) AS 'wins'
from games where ((home=(?) AND visitor=(?)) OR (home=(?) AND
visitor=(?))) group by winner order by wins desc');
$sth->execute( $team1,$team2,$team2,$team1) || die $dbh->errstr;
> the whole CFFI bindings are more portable. As for any advantage over
> UFFI, uh, I am the wrong person to ask, but IIUC CFFI is more
> efficient than UFFI under certain implementations. Apologies now if
> that is gross misinformation, but to me the only thing that matters is
What is the easiest way to follow up on this? In particular, UFFI is
often grossly inefficient on CMUCL, as it doesn't propagate the
appropriate type information down to the alien calls it translates
into. It's enough of a problem that UFFI is essentially unusable if
you want to call out to C in a tight inner loop --- you'll cons more
than 100KB every time through the call.
rif
CL-USER 2 > (asdf:oos 'asdf:load-op :clsql)
; [snip]
NIL
CL-USER 3 > (use-package :clsql)
T
CL-USER 4 > (connect '("orcl" "edi" "xxx") :database-type :oracle)
; [snip]
#<CLSQL-ORACLE:ORACLE-DATABASE orcl/edi OPEN 2069FB34>
CL-USER 5 > (execute-command "create table games (id number primary key, home varchar2(10), visitor varchar2(10), winner varchar2(10))")
CL-USER 6 > (defun clsql-thing-as-string (thing)
(with-output-to-string (clsql-sys::*sql-stream*)
(clsql-sys::output-sql thing *default-database*)))
CLSQL-THING-AS-STRING
CL-USER 7 > (defun query* (fmt-string &rest args)
(query (apply #'format nil fmt-string (mapcar #'clsql-thing-as-string args))))
QUERY*
CL-USER 8 > (defun execute-command* (fmt-string &rest args)
(execute-command (apply #'format nil fmt-string (mapcar #'clsql-thing-as-string args))))
EXECUTE-COMMAND*
CL-USER 9 > (defun insert-game (id home visitor)
(execute-command* "insert into games (id, home, visitor, winner) values (~A, ~A, ~A, ~A)"
id home visitor (if (oddp (random 2)) home visitor)))
INSERT-GAME
CL-USER 10 > (dotimes (i 30) (insert-game i "FOO" "BAR"))
NIL
CL-USER 11 > (query "select * from games")
((0 "FOO" "BAR" "FOO") (1 "FOO" "BAR" "FOO") (2 "FOO" "BAR" "BAR") (3 "FOO" "BAR" "BAR") (4 "FOO" "BAR" "BAR") (5 "FOO" "BAR" "FOO") (6 "FOO" "BAR" "BAR") (7 "FOO" "BAR" "FOO") (8 "FOO" "BAR" "BAR") (9 "FOO" "BAR" "FOO") (10 "FOO" "BAR" "FOO") (11 "FOO" "BAR" "FOO") (12 "FOO" "BAR" "BAR") (13 "FOO" "BAR" "FOO") (14 "FOO" "BAR" "BAR") (15 "FOO" "BAR" "BAR") (16 "FOO" "BAR" "FOO") (17 "FOO" "BAR" "FOO") (18 "FOO" "BAR" "FOO") (19 "FOO" "BAR" "BAR") (20 "FOO" "BAR" "FOO") (21 "FOO" "BAR" "BAR") (22 "FOO" "BAR" "FOO") (23 "FOO" "BAR" "FOO") (24 "FOO" "BAR" "FOO") (25 "FOO" "BAR" "FOO") (26 "FOO" "BAR" "BAR") (27 "FOO" "BAR" "BAR") (28 "FOO" "BAR" "BAR") (29 "FOO" "BAR" "FOO"))
("ID" "HOME" "VISITOR" "WINNER")
CL-USER 12 > (defun winner-table (team1 team2)
(query* "select winner as team, count(winner) as wins from games where ((home = ~A AND visitor = ~A) OR (visitor = ~2:*~A AND home = ~A)) group by winner order by wins desc"
team1 team2))
WINNER-TABLE
CL-USER 13 > (winner-table "FOO" "BAR")
(("FOO" 17.0) ("BAR" 13.0))
("TEAM" "WINS")
CL-USER 14 > (loop for i from 30 to 40 do (insert-game i "FOO" "BAR"))
NIL
CL-USER 15 > (winner-table "FOO" "BAR")
(("FOO" 26.0) ("BAR" 15.0))
("TEAM" "WINS")
rif wrote:
> Kenny Tilton <kti...@nyc.rr.com> writes:
>
>
>>the whole CFFI bindings are more portable. As for any advantage over
>>UFFI, uh, I am the wrong person to ask, but IIUC CFFI is more
>>efficient than UFFI under certain implementations. Apologies now if
>>that is gross misinformation, but to me the only thing that matters is
>
>
> What is the easiest way to follow up on this?
http://common-lisp.net/project/cffi/
Look for the cffi-luis branch (I do not darc, so I grab the latest
tarball from the link for those). CMUCL is listed as "working", so I
think you could cobble together a timing test from those in the
regression test suite.
And the three projects i'm interested in : Verrazano,CFFI
and Hello-C are doing good. Right Kenny?
It looks that i'm one lucky bastard. :)
Slobodan
fireblade wrote:
> Eric Lavigne wrote:
>
>>So this brings it to 6 out of 9?
>>Eric
>
>
> And the three projects i'm interested in : Verrazano,CFFI
> and Hello-C are doing good. Right Kenny?
Hello-C (UFFI plus callbacks) is deprecated. I do have a Hello-CFFI
package, but that uses CFFI and adds some dorky little FF utilities I
like, and in time I will suggest them to the CFFI crew to see if they
will adopt them so I can Just Use CFFI.
> It looks that i'm one lucky bastard. :)
It looks /like/ we are all lucky. :)
> > > One project retired in the face of unexpected difficulties,
> >
> > Which one was that?
> >
> > Matt
>
> The retired project is PLT-DB. That student announced before the Sept 1
> deadline that there was not enough time remaining, and that he was
> giving up.
Google SOC is a great way for students to realize that summer is way
too short.
:-)
Andras
I used to find this was a huge problem for me, and ended up replacing
uffi:with-foreign-objects in a way which was incompatible with other
uffi projects, for my personal use. (and added a pointer-to-object
macro.)
As soon as I have time, I plan to see how this works out in CFFI. It
should make game implementation in CL with CMU/SBCL a lot more feasible.
Cheers.
--
Julian Squires
Thanks Edi! I'm going to find that very useful.
> On Thu, 22 Sep 2005 15:28:27 +0200, <spam...@agharta.de> wrote:
>>
>> CL-USER 7 > (defun query* (fmt-string &rest args)
>> (query (apply #'format nil fmt-string (mapcar #'clsql-thing-as-string args))))
>
> Thanks Edi! I'm going to find that very useful.
Note that this was just a quick hack for the demo - you'll lose the
ability to feed keyword arguments to QUERY and EXECUTE-COMMAND. But I
think using something like CLSQL-THING-AS-STRING is generally a good
idea if, like in Perl, you are using "strings with gaps" to
communicate the SQL commands to the database server.
Cheers,
Edi.
Yes, it is a nice extension method. It gave me insight into how I can
match perl string ticks with CL, which is far, far easier than using
perl to match lisp.
Yeah , hardly waiting for some spare time to try what Verrazano and
CFFI
are capable of .
slobodan
http://blazeski.blogspot.com/