Ann: scalable websocket based chat server in SWI-Prolog
Jan Wielemaker
I've done a little bit of hacking to see whether SWI-Prolog's
infrastructure was powerful enough to implement a scalable chatroom in
Prolog. It was a bit of a challenge, but the result is a very simple
chatroom build on top of a library that can deal with managing many
websockets with few resources. A chatroom consists of
- A message queue on which all events and control messages from all
websockets are concentrated.
- A predicate to send a message to a specific client
- A predicate to broadcast a message to all members of the room
The code is (for now) hosted on GitHub at
https://github.com/JanWielemaker/swi-chat
It consists of the manager library (chatroom.pl), the chat application
(demo_chat.pl) and a stress-testing client library (stress_client.pl).
Tested with 300 clients sending messages at random times. The client
code is a bit simple minded, and it became clear that the clients would
get stressed sooner than the server :-)
As yet, only tested with Linux as server, Firefox, Prolog and Chromium
as client. I *think* the server should also work on Windows.
**Requires SWI-Prolog 7.1.19 or later**
Enjoy --- Jan
Anne Ogborn
(and, fwiw, anybody who wants a great education in how to write SWI-Prolog web applications,
read the code!)
Tested running locally on win 7 64 bit, on 7.1.19 devel 64 bit,
works with Chrome Version 36.0.1985.125 m , Firefox 26, and IE 11.0
IE 9 is broken because it doesn't support websockets.
Torbjörn Lager
BTW, perhaps now it would be worthwhile to implement XMPP and/or IRC on top of library(space)? (Don't know how much work is required for that though.)
--
You received this message because you are subscribed to the Google Groups "SWI-Prolog" group.
To unsubscribe from this group and stop receiving emails from it, send an email to swi-prolog+unsubscribe@googlegroups.com.
Visit this group at http://groups.google.com/group/swi-prolog.
For more options, visit https://groups.google.com/d/optout.
--
Torbjörn Lager
Professor of General and Computational Linguistics
Department of Philosophy, Linguistics and Theory of Science
University of Gothenburg
Box 200, SE-405 30 Gothenburg, Sweden
Phone: +46317864962
Jan Wielemaker
> Maybe library(chatroom) should be given another and more abstract name
> eventually, since it provides a functionality that is very useful in
> many other circumstances too, not just for chat? May I suggest
> library(space)? That's more abstract and it's also the case that a room
> is a kind of space.
problem. Library(space) is out, because we already have that, containing
spatial reasoning support. I used this name to refer to a chatroom as a
general concept: a room where communication lines come together and data
can be exchanged. I'm happy with another name, possibly derived from a
library serving a similar role.
> BTW, perhaps now it would be worthwhile to implement XMPP and/or IRC on
> top of library(space)? (Don't know how much work is required for that
> though.)
and possibly react on the events.
> I also started thinking about how pengines and library(space) might fit
> together. The notion of a _workspace_ comes to mind. It should be
> possible I suppose to allow a pengine to enter into/be created in a
> workspace and for other people (or pengines) in that space to "chat"
> with it (i.e. send it requests and listen for reponses)? If this makes
> sense, and I don't see why not, then that seems to provide exactly what
> is needed for an IDE for collaborative Prolog programming. One of more
> Prolog developers meet with one or more pengines in a workspace equipped
> with editors, debuggers and what have you. Jan, I know that an web-based
this stuff is mostly orthogonal. Pengines can read the current shared
state to answer queries. They could also modify the state, either
directly or by sending a message to the chatroom/space event queue.
By themselves they are not very well equipped to listen to events from
a shared space because they are more or less constrained to be slaves
of a single master that understands the pengine's state machine.
In an shared IDE scenario, chatroom/space can either extend or replace
TogetherJS in making multiple users cooperate. It can easily keep two
CodeMirror editors in sync because a CodeMirror editor emits change
events and can process change events. It can also distribute pengine
answers to queries over multiple browsers, etc.
Anyway, these are mostly open issues ... The library does make the
implementation of servers that need to deal with a shared workspace,
mixed initiative or low-latency synchronization nearly trivial.
Cheers --- Jan
Torbjörn Lager
> I also started thinking about how pengines and library(space) might fit> IDE is something that is high on your agenda. Is this how you have
> together. The notion of a _workspace_ comes to mind. It should be
> possible I suppose to allow a pengine to enter into/be created in a
> workspace and for other people (or pengines) in that space to "chat"
> with it (i.e. send it requests and listen for reponses)? If this makes
> sense, and I don't see why not, then that seems to provide exactly what
> is needed for an IDE for collaborative Prolog programming. One of more
> Prolog developers meet with one or more pengines in a workspace equipped
> with editors, debuggers and what have you. Jan, I know that an web-based
> thought about it, or do you have another idea?I have not really thought about the relation. I think that pengines and
this stuff is mostly orthogonal. Pengines can read the current shared
state to answer queries. They could also modify the state, either
directly or by sending a message to the chatroom/space event queue.
By themselves they are not very well equipped to listen to events from
a shared space because they are more or less constrained to be slaves
of a single master that understands the pengine's state machine.
Anne Ogborn
I sure can!
Some things I fantasize about having, (mostly without the commitment of writing them!):
* A library of common web design patterns.
The web framework already demonstrates that reasoning can be a powerful tool for simplifying the task of defining web applications.
In other, 'dumb' languages, application developers are forced to largely write their application with a lexical order that corresponds to page load order. Indeed, PHP builds this idea into the language syntax.
By contrast, a logic language builds up the page as a grammar, an activity that is not inherently left-to-right. Further, in a declarative language we have a new abstraction - we can declare what should happen, and often arrange the underlying library to just 'make it do the right thing', even when that's complex, or happens in several distant places.
I have a phrase I use a lot, 'sweeping it under the rug'. To me, much of advance in software engineering is about finding ways to reduce the number of concerns for the application programmer - to 'sweep under the rug' as many issues as we can.
SWI-Prolog (and declarative languages in general) are amazing at doing this. The resource management facility is an example. If I write a widget that does some special menu, you can just use it as an inclusion without knowing or caring that it depends on some css file.
::Annie is here, standing on a lumpy rug::
Consider the conceptually simple task of adding a single field to the user table of a web app, and some ui that lets the user modify that field.
This involves a lot of pieces, scattered throughout the application code:
modify Android and iphone UI
modify the html and css files, and possibly create new resources, possibly for several different form factors.
build a new ajax or other feed for the data.
modify the database schema. If the new field is an image, build a scheme for uploading images.
build a UI to modify the field
Weblog ( https://github.com/Anniepoo/weblog )is a modest and somewhat experimental attempt to build a library of tools that make such things simpler, a conspiracy of Wouter, Thanos, and I. It started as a collection of some common bits of useful web stuff that I got tired of disentangling each time I wanted them. For a while I was trying to add all the Welie patterns, but at some point realized many of them were as well provided in straight javascript. I'm now concentrating on those that have the most widely distributed effects, and most benefit from Weblog's 'under the covers' approach. Of course such things are precisely those that are hardest to build!
Examples of some things that could be packaged up:
user identity
wikification
comments
pagination
ecommerce checkout
various feeds - google maps, books, etc. Some are in weblog
* A declarative system for establishing mappings between the DOM and Prolog.
Most of the time web elements are representations of data. So why can't we build something that declaratively expresses that representation?
I'm imagining some world where I can write some selector like syntax, an operator, and a prolog term, and have the html stay synced with the prolog, and if it's an input tag, the prolog stay synced with the value.
I've been noodling around with what that would look like, but I'm feeling like it's deep breathing for little Annie.
&:^[ &8vC &:->
&8c) *8cD &8U) <-- Annie, surrounded by bulgy brained people on SWI-Prolog list
&B-/ &:=] &:-7
* mobile support
Before we can really play in the web dev space some solution for this is called for.
>Maybe library(chatroom) should be given another and more abstract name eventually, since it provides a functionality that is very useful in many other circumstances too, not just for chat? May I suggest library(space)? That's more abstract and it's also the case that a room is a kind of space.
Agreed! I'm using library(chatroom) to write a shared node-and-arc drawing tool this weekend.
>BTW, perhaps now it would be worthwhile to implement XMPP and/or IRC on top of library(space)? (Don't know how much work is required for that though.)
Probably best as a pack. Might be a good project for one of my students this fall.
Similarly, the chatroom wrapper might make a good pack.
The IRC protocol is dead simple, and quite generally useful. Arguably it'd be best practice to replace Jan's little back end with something based on it.
I'd immediately use it to make a space that could echo ##prolog into a place we could work with students.
Another widely useful application would be a presence library. This would present a list of people in the room, control muting and banning, handle IP bans, etc. etc.
I also started thinking about how pengines and library(space) might fit together. The notion of a _workspace_ comes to mind. It should be possible I suppose to allow a pengine to enter into/be created in a workspace and for other people (or pengines) in that space to "chat" with it (i.e. send it requests and listen for reponses)?
How's this work with the notion of an application?
beyond the request/response model, I'd invite considering websockets. Forever saddling us with client-must-initiate is a heavy penalty.
And is it reasonable for pengine IO to be through websockets?
I know you're working towards an online teach prolog solution, but would suggest you're actually building something very fundamental and powerful. Consider a wide array of use cases.
>If this makes sense, and I don't see why not, then that seems to provide exactly what is needed for an IDE for collaborative Prolog programming. One of more Prolog developers meet with one or more pengines in a workspace equipped with editors, debuggers and what have you. Jan, I know that an web-based IDE is something that i high on your agenda. Is this how you have thought about it, or do you have another idea?
Torbjorn, I think it's what's needed for many tasks. If web 2.0 was about social media, web 3.0 is perhaps about collaborative desktop like tools like google docs, OneDrive, etc and about shared environments (turntable.fm, IRC, Second Life, World of Warcraft). We aren't going to replace PHP as what people build shopping cart systems with.
But in a couple years I want to start meeting kids who sneer at Clojure and think all the cool kids are doing Prolog. That means we need to start making things that are hard to make any other way. We do that already, but few people understand them or get that they're cool.
Fortunately, some of us do! 8cD
Anne Ogborn
this stuff is mostly orthogonal. Pengines can read the current shared
state to answer queries. They could also modify the state, either
directly or by sending a message to the chatroom/space event queue.
By themselves they are not very well equipped to listen to events from
a shared space because they are more or less constrained to be slaves
of a single master that understands the pengine's state machine.
I must confess, it took me a while to get my head around the importance of the fact that a pengine slave has a unique master. And I'm wondering what all it buys us.
They may be orthogonal, but they aren't without synergy. We're moving towards an environment in which we have good tools for doing computation where the data is, and doing UI where the human is, without the last 20 years of hooting and hollering to do it.
(since 'space' is taken as a name, how about 'collaborate' ?)
Torbjörn Lager
> I can't think of anything else that needs to be there.
I sure can!
Some things I fantasize about having, (mostly without the commitment of writing them!):
* A library of common web design patterns.
The web framework already demonstrates that reasoning can be a powerful tool for simplifying the task of defining web applications.
In other, 'dumb' languages, application developers are forced to largely write their application with a lexical order that corresponds to page load order. Indeed, PHP builds this idea into the language syntax.
By contrast, a logic language builds up the page as a grammar, an activity that is not inherently left-to-right. Further, in a declarative language we have a new abstraction - we can declare what should happen, and often arrange the underlying library to just 'make it do the right thing', even when that's complex, or happens in several distant places.
Instead of telling Prolog, like in the old days, to use writeln/1 in order to write "Hello world" output to the TTY, you tell your pengine to send {"event": "output", "id":"289-94..", "data": "Hello world"} to the browser. Better still, as a Prolog programmer, you don't even have to be able to _read_ the JSON that calling the goal pengines_output("Hello world") produced. You could hire yourself a good web front-end programmer that under your supervision will see to it that your "Hello world" ends up in the right place at the right time, maybe written (in any colour you might choose!) on a button or in a menu, or maybe spoken by your favourie speeh synthesizer. I like the purity of this a lot. Granted, it saves only the _rich_ Prolog programmer from having to deal with HTML, CSS and JavaScript. The rest of us just have to bite that bullet. If you're poor, knowing some HTML, CSS and JavaScript is the only way to be productuve in the web space. If you're a commerical company or the happy receiver of a lot of R&D funding, it is probably better to stick to Prolog and hand the rest over to people that make a living out of turning JSON into web application behaviour.
I have a phrase I use a lot, 'sweeping it under the rug'. To me, much of advance in software engineering is about finding ways to reduce the number of concerns for the application programmer - to 'sweep under the rug' as many issues as we can.
SWI-Prolog (and declarative languages in general) are amazing at doing this. The resource management facility is an example. If I write a widget that does some special menu, you can just use it as an inclusion without knowing or caring that it depends on some css file.
::Annie is here, standing on a lumpy rug::
Consider the conceptually simple task of adding a single field to the user table of a web app, and some ui that lets the user modify that field.
This involves a lot of pieces, scattered throughout the application code:
modify Android and iphone UI
modify the html and css files, and possibly create new resources, possibly for several different form factors.
build a new ajax or other feed for the data.
modify the database schema. If the new field is an image, build a scheme for uploading images.
build a UI to modify the field
Weblog ( https://github.com/Anniepoo/weblog )is a modest and somewhat experimental attempt to build a library of tools that make such things simpler, a conspiracy of Wouter, Thanos, and I. It started as a collection of some common bits of useful web stuff that I got tired of disentangling each time I wanted them. For a while I was trying to add all the Welie patterns, but at some point realized many of them were as well provided in straight javascript. I'm now concentrating on those that have the most widely distributed effects, and most benefit from Weblog's 'under the covers' approach. Of course such things are precisely those that are hardest to build!
Examples of some things that could be packaged up:
user identity
wikification
comments
pagination
ecommerce checkout
various feeds - google maps, books, etc. Some are in weblog
* A declarative system for establishing mappings between the DOM and Prolog.
Most of the time web elements are representations of data. So why can't we build something that declaratively expresses that representation?
I'm imagining some world where I can write some selector like syntax, an operator, and a prolog term, and have the html stay synced with the prolog, and if it's an input tag, the prolog stay synced with the value.
I've been noodling around with what that would look like, but I'm feeling like it's deep breathing for little Annie.
* mobile support
Before we can really play in the web dev space some solution for this is called for.
>Maybe library(chatroom) should be given another and more abstract name eventually, since it provides a functionality that is very useful in many other circumstances too, not just for chat? May I suggest library(space)? That's more abstract and it's also the case that a room is a kind of space.
Agreed! I'm using library(chatroom) to write a shared node-and-arc drawing tool this weekend.
>BTW, perhaps now it would be worthwhile to implement XMPP and/or IRC on top of library(space)? (Don't know how much work is required for that though.)
Probably best as a pack. Might be a good project for one of my students this fall.
Similarly, the chatroom wrapper might make a good pack.
The IRC protocol is dead simple, and quite generally useful. Arguably it'd be best practice to replace Jan's little back end with something based on it.
I'd immediately use it to make a space that could echo ##prolog into a place we could work with students.
Another widely useful application would be a presence library. This would present a list of people in the room, control muting and banning, handle IP bans, etc. etc.
I also started thinking about how pengines and library(space) might fit together. The notion of a _workspace_ comes to mind. It should be possible I suppose to allow a pengine to enter into/be created in a workspace and for other people (or pengines) in that space to "chat" with it (i.e. send it requests and listen for reponses)?
How's this work with the notion of an application?
beyond the request/response model, I'd invite considering websockets. Forever saddling us with client-must-initiate is a heavy penalty.
And is it reasonable for pengine IO to be through websockets?
I know you're working towards an online teach prolog solution, but would suggest you're actually building something very fundamental and powerful. Consider a wide array of use cases.
>If this makes sense, and I don't see why not, then that seems to provide exactly what is needed for an IDE for collaborative Prolog programming. One of more Prolog developers meet with one or more pengines in a workspace equipped with editors, debuggers and what have you. Jan, I know that an web-based IDE is something that i high on your agenda. Is this how you have thought about it, or do you have another idea?
Torbjorn, I think it's what's needed for many tasks. If web 2.0 was about social media, web 3.0 is perhaps about collaborative desktop like tools like google docs, OneDrive, etc and about shared environments (turntable.fm, IRC, Second Life, World of Warcraft). We aren't going to replace PHP as what people build shopping cart systems with.
But in a couple years I want to start meeting kids who sneer at Clojure and think all the cool kids are doing Prolog. That means we need to start making things that are hard to make any other way. We do that already, but few people understand them or get that they're cool.
Torbjörn Lager
--
You received this message because you are subscribed to the Google Groups "SWI-Prolog" group.
To unsubscribe from this group and stop receiving emails from it, send an email to swi-prolog+...@googlegroups.com.
Visit this group at http://groups.google.com/group/swi-prolog.
For more options, visit https://groups.google.com/d/optout.
Torbjörn Lager
Anne Ogborn
>Of course, another problem with this kind of scenario is that it creates what linguists would refer to as a "turn-taking problem". How do we prevent two pengines from "talking" at the same time? Or a pengine from "interrupting" a human participant (think of a speech interface to the chat app and it becomes very clear that it's tricky). In multi-party dialogues this is a normal thing that we as humans are able to handle very well. Pengines may have more trouble though.
This problem is inherently domain specific.
Consider a social robot. The robot interprets an utterance as funny. It triggers a laugh animation, and makes an appropriate reply. Oops- it's got to do the laugh first, and then the reply. This is the core headache of making a social robot.
I'd point out that pengines are talking by message passing, so they're going to queue the messages anyway.
Anne Ogborn
(I spent 4 years writing mostly LSL).
To me 'every message is a broadcast' is a normal way of life.
Anne Ogborn
Angular is interesting, and is indeed towards what I was driving at, but has the usual feel of declarative stuff shoehorned in to fit with a borked language. However, I'm willing to give a little to a library with a release callecd 'chimeric-glitterfication'.
>However, a web _application_ (as opposed to a web page) also has _behaviour_ that needs to be catered for. Traditionally, the behaviour of a web application is a job passed over to JavaScript, and indeed, in his http/html_* libraries Jan again choose to be very practical (for good reasons I would say), and allows the programmer/grammar writer to include JavaScript in generated document by means of the quasi quotation mechanism.
Yes, Jan's wisely avoided the whole issue. But I'm suggesting a layer above that does approach this issue of getting a declarative relation between the DOM and the prolog database.
Ignoring a large raft of issues for the moment, here's a couple of approaches:
first approach
Most javascript is just used to cover the impedence bump between the data and the contents of the DOM.
So I'm imagining some DSL with syntax vaguely like
selector <**> query
where selector is a jquery or CSS selector looking thing, and where query is a query on the server, and which maps the solutions to selector to the proofs of term. The operator <**> asserts that changes in either 'end' will be propagated through to the other end, and there are single implication versions <** and **>
query is a Prolog term, but with a few caveats
First, we can share variables across the <**> divide
#employees !! DIV .employee-number .html(X) <** employee_number(X)
Second, !! is a marker showing where to make a new element, - imagine the case of a row per employee ID, with fields for name, position, etc.
Third, there probably needs to be some page and session spaces, perhaps special form modules page: and session:
and a bunch of other stuff that I have or haven't thought of.
And how we monitor for change on the prolog side, I have no idea.
second approach
One includes some javascript code, and provides a (probably pengine) handler (or this gets handled under the covers by an inclusion).
When the page is served a copy of the DOM tree is retained on the server.
When the DOM tree on the client changes, it reports the change to the server. When the prolog program modifies the server side DOM, the pengine tells the client to update it's DOM to look like the server.
There is presumably some hook for querying goals when the server side DOM changes.
Under the covers this could work like git - compute a hash on each primitive, and hash the product of the hashes of it's children and it's attributes for each container node, working bottom to top.
>This is tricky stuff. Some behaviour may be possible to capture declaratively, but I personally tend to believe that a lot cannot be captured declaratively, at least not in a natural and maximally flexible way, and not if it is going to appeal to the masses. (And we want out Prolog platforms to appeal to the masses, don't we. ;-))
I disagree - I think we can specify what we want, and I believe there's people on this list who can figure out how to implement it.
>(And we want out Prolog platforms to appeal to the masses, don't we. ;-))
I don't know what others want, but I'd love to have my choice of well paying jobs doing Prolog on interesting project domains, working with people with actual people skills. I sometimes suspect people prefer Prolog remain obscure.
>You could hire yourself a good web front-end programmer that under your supervision will see to it that your "Hello world" ends up in the right place at the right time,
Uh, I haven't had one of those since some time in the late 1990's
And while I'm certainly not as good at such things as Jessica, the web designer who designed the new SWI-Prolog site, I do know enough css and html to get by, and hope other back end people would.
I really don't think we should design to ensure job security for front end people. Instead we should be doing what we do with other ugly but non-removable things like assembler languages - treat them with higher abstractions and usually forget they're there.
(For a while I told people my religion forbade me to make web pages - you know, Jews and Muslims won't eat pork, I don't do web pages).
>
Yes, but only as long as you're happy with the CSS in the file. Suppose you want to change the size of a text field, or the colour of "Hello world", then it might become messy.
Well, then you cascade the style sheet. I prefer an incremental approach to development, and part of that is that things come up without a fuss. I think styling is a pretty solved problem.
The important thing is that the default behavior is a whiteboard that looks and acts like some default whiteboard 'out of the box'. Serving a 500 or 404 page is not a good default.
As Jeremiah 4:1 says "If you, Israel, will return, then return to me," declares the Lord.
See, even Jeremiah was a declarative programmer, and he had to get the Lord's help to deal with imperative systems.
;-)
::Annie is here, reading the bible. On closer inspection, it's a copy of Craft of Prolog::
(appropriately I have to finish this now and turn off computer, as there's a terrific lightning storm going on).
Jan Wielemaker
As far as I see it now, Pengines are not agents. A sensible agent can
listen to the world, pick messages from diverse sources in the world,
maintain beliefs about the world and react on that.
Pengines listen to a queue and its `master' must go through a stateful
protocol not to upset the pengine. They basically serve RPC calls with
the added beauty that you can hand them a program so you get/send the
data in the format you (= client) wants to manage it. That is a very
useful building block, but not an agent. To a certain extend, based on
I/O, you can have a `conversation' with an pengine, but this too is
limited in the sense that the pengine decides when to read and write.
Using I/O gives the initiative to the pengine instead of his master.
You could add a pengine to a chatroom by somehow connecting
pengine_input/2 to listen for events in a chatroom and act by using the
chatrooms (output) communication predicates. This is not trivial because
SWI-Prolog's threading infrastructure does not make it easy to broadcast
a message to a set of threads about which you have no knowledge.
Possibly we should add a broadcasting facility. The POSIX solution for
that are _condition variables_, but that implies using a mutex and I'd
like to keep these things as far as possible from the Prolog user
because it is too easy to get a deadlock. The Ciao alternative is that a
thread can block on a dynamic predicate and wakes (providing the next
answer) if another thread performs an assertz/1 on this predicate. That
might be more Prolog minded, although it violates the logical update
view. Another option might be a thread_get_message/3 option to wait for
a queue without removing the message (i.e., a blocking peek_message).
Opinions?
Connecting a Pengine like that would allow an external process to add
active components (agents) to the chatroom that can both communicate
with the creator (master) using pengine_output/1 and other participants
of the chatroom. You can probably do funny things with that, but most
likely it only works if everyone wants to cooperate faithfully :-(
Cheers --- Jan
Jan Wielemaker
> (since 'space' is taken as a name, how about 'collaborate' ?)
library(http/collaboration). It is a bit long word. Do we have
something shorter in that area? Another option might be "workspace"?
I'd assume people will start looking for workspace when looking for
a library like this, no? The only problem is that the library
actually takes care of the _communication_ involved in building
a shared workspace. The workspace itself is not part of it (and
should not become part of it either). In that sense "collaboration"
is more about communication.
Cheers --- Jan
Jan Wielemaker
>> I can't think of anything else that needs to be there.�
>
> I sure can!
A lot of interesting ideas!
Cheers --- Jan
Torbjörn Lager
library(http/communicator)
library(http/shared_space)
library(http/hub)
library(http/commhub)
library(http/channel)
Torbjörn Lager
As far as I see it now, Pengines are not agents. A sensible agent can
listen to the world, pick messages from diverse sources in the world,
maintain beliefs about the world and react on that.
An agent is a
- transition system (i.e. a system of states and transitions between them) that is
- interactive (i.e. responds to external stimuli by change of state)
- autonomous (i.e. is able to change state without external stimuli, perhaps in response to an internal event)
- adaptable (i.e. is able to change the transitions by which is changes states).
- teleological (i.e. has goals, which it pursues through its actions)
Pengines listen to a queue and its `master' must go through a stateful
protocol not to upset the pengine.
They basically serve RPC calls with
the added beauty that you can hand them a program so you get/send the
data in the format you (= client) wants to manage it.
That is a very
useful building block, but not an agent. To a certain extend, based on
I/O, you can have a `conversation' with an pengine, but this too is
limited in the sense that the pengine decides when to read and write.
Using I/O gives the initiative to the pengine instead of his master.
Torbjörn Lager
Another story, somewhere in between, is exemplified by the expert system demo:
ME: yes
ME: yes
ME: next % means: can you prove that in another way?
ME: yes
You could add a pengine to a chatroom by somehow connecting
pengine_input/2 to listen for events in a chatroom and act by using the
chatrooms (output) communication predicates. This is not trivial because
SWI-Prolog's threading infrastructure does not make it easy to broadcast
a message to a set of threads about which you have no knowledge.
Possibly we should add a broadcasting facility. The POSIX solution for
that are _condition variables_, but that implies using a mutex and I'd
like to keep these things as far as possible from the Prolog user
because it is too easy to get a deadlock. The Ciao alternative is that a
thread can block on a dynamic predicate and wakes (providing the next
answer) if another thread performs an assertz/1 on this predicate. That
might be more Prolog minded, although it violates the logical update
view. Another option might be a thread_get_message/3 option to wait for
a queue without removing the message (i.e., a blocking peek_message).
Opinions?
Connecting a Pengine like that would allow an external process to add
active components (agents) to the chatroom that can both communicate
with the creator (master) using pengine_output/1 and other participants
of the chatroom. You can probably do funny things with that, but most
likely it only works if everyone wants to cooperate faithfully :-(
Jan Wielemaker
> Pengines listen to a queue and its `master' must go through a stateful
> protocol not to upset the pengine.
>
>
> I think "upset" is too strong a word. It doesn't crash, nor does it move
> into an inconsistent state. In fact it doesn't change states at all.
thread can ask it something. The other can walk away with the next
answer by asking for the next, but it doesn't know the question. Brings
the hitchhikers guide to the galaxy to mind :-)
> What you get back from a pengine if you make a request that is outside
> the protocol is a protocol_error - the pengine's way of telling you it
> didn't understand you. You would know you have to say something else in
> order to move the conversation forward. Maybe the protocol error event
> should contain the list of command that the pengine _will_ understand in
> its current state? From state 2 it would send error(ID,
> error(protocol_errror, [destroy, ask])) and from state 6 it would send
> error(ID, error(protocol_error, [next, stop, ask, destroy])). If so, the
> protocol error should perhaps be an error of a special _kind_, (i.e.
> with their own handlers - onperror maybe.
programming errors. We might want to add a clue about the current state
to simplify debugging, but I don't see that much point adding more
boilerplate to fomalise that. What is the point of asking `ask`, getting
back that there is an open query, so ask `next` to get the next answer
to who-knows-which-query?
Cheers --- Jan
Torbjörn Lager
On 08/11/2014 03:08 PM, Torbjörn Lager wrote:True. But chatting with two `thingies' to a pengine is meaningless. One
Pengines listen to a queue and its `master' must go through a stateful
protocol not to upset the pengine.
I think "upset" is too strong a word. It doesn't crash, nor does it move
into an inconsistent state. In fact it doesn't change states at all.
thread can ask it something. The other can walk away with the next
answer by asking for the next, but it doesn't know the question. Brings
the hitchhikers guide to the galaxy to mind :-)
I don't see compelling reasons to do that. Protocol errors are just yourWhat you get back from a pengine if you make a request that is outside
the protocol is a protocol_error - the pengine's way of telling you it
didn't understand you. You would know you have to say something else in
order to move the conversation forward. Maybe the protocol error event
should contain the list of command that the pengine _will_ understand in
its current state? From state 2 it would send error(ID,
error(protocol_errror, [destroy, ask])) and from state 6 it would send
error(ID, error(protocol_error, [next, stop, ask, destroy])). If so, the
protocol error should perhaps be an error of a special _kind_, (i.e.
with their own handlers - onperror maybe.
programming errors. We might want to add a clue about the current state
to simplify debugging, but I don't see that much point adding more
boilerplate to fomalise that. What is the point of asking `ask`, getting
back that there is an open query, so ask `next` to get the next answer
to who-knows-which-query?
I think you are right. Scrap that.
Anne Ogborn
----- Forwarded Message -----
From: Anne Ogborn <anni...@yahoo.com>
To: Jan Wielemaker <J.Wiel...@vu.nl>
Cc:
Sent: Monday, August 11, 2014 9:40 AM
Subject: Re: [SWIPL] Ann: scalable websocket based chat server in SWI-Prolog
>As far as I see it now, Pengines are not agents. A sensible agent can
listen to the world, pick messages from diverse sources in the world,
maintain beliefs about the world and react on that.
with the ability to add private predicates to the knowledge set.
Anne Ogborn
'whiteboard' is the obvious name for a whiteboard. But 'sort' is about sorting.
If you need to nounify it you could use _lib.
But I'm OK with the word length.
Alternatives: collude and fraternize have negative implications.
partners is nice but non obvious. Allies is similarly nonobvious.
Workspace, at least in US english, is really is more about shared 'place', and has
other, useful meanings. I'd avoid it.
So I land on collaboration as best of an imperfect world.
Anne Ogborn
If we can live with discontiguous (I have to look that spelling up every time I use it) we can
live with 13 chars in collaboration.
8cD
::Annie is here, typing in DCG's for a language with keywords like onomatopoeia, apoptosis and prestidigitation.::
Anne Ogborn
Martin Sondergaard
I have written a chatbot, and I'd like to get it to appear
in Second Life in some form.
I don't know if this would be difficult.
--
Martin Sondergaard,
London, UK.
Anne Ogborn
> However, a web _application_ (as opposed to a web page) also has _behaviour_ that needs to be catered for. Traditionally, the behaviour of a web application is a job passed over to JavaScript, .. A practical solutions, but hardly a step towards a declarative way to describe the behaviour of web applications. This is tricky stuff. Some behaviour may be possible to capture declaratively, but I personally tend to believe that a lot cannot be captured declaratively, at least not in a natural and maximally flexible way, and not if it is going to appeal to the masses. (And we want out Prolog platforms to appeal to the masses, don't we. ;-))
I think there's plenty of support- and lots of existing attempts at the problem - for sweeping javascript, html, css, and the whole fetid mass under the rug.
The problem is, these attempts invariably have tried to retain the imperative nature of Javascript under the hood. That's a suspicious activity. The browser side is a naturally declarative place - witness HTML and CSS, and that the most successful of the attempts to deal with javascript, jquery, is declarative at it's core.
As for the masses, I think anybody interested in our back end will be OK with a selector based system. What the masses accept is a matter of education.
>Enter pengines! You see, I happen to think that pengines (or something similar but even more refined) can help here.
Well, you can't just replace all javascript based behavior with pengines, simply because the pengine has a round trip to the server. If you really just want Prolog on the browser, look at Jekejeke or at Raivo Laanemet's Prolog in Javascript.
But I'm not sure Prolog is the best declarative language for the browser anyway.
>I'm not saying this is something that logic can't deal with (I'm sure you can deal with some of it using e.g. event calculus), but will it become natural?
For some things, probably not. Taking the SWI-Prolog website as an example, the javascript that handles the menus across the top is probably best just javascript. But populating the tree control on the left is something that could be handled by a declarative system, and certainly things like my name or 'login' prompt and 'powered by ' just need mapped to a variable.
In theory, you could always have a hidden field with an onchange listener as an interface between javascript and declarative world, but in practice I suspect an escape will be less elegant but more practical.
>I don't want to completely rule out server side generation of pages and the use of quasi quotation though.
I don't think we'll be completely getting away from the ajax pattern of making a DOM fragment on the server and sending to the client to replace something any time soon, either.
>They may still have a role to play when it comes to the problem of building reusable components for the web.
>since dynamically generated HTML for components may well _include_ pengines serving JSON to JavaScript which performs the necessary DOM manipulations in order to create behaviour. Makes me a bit dizzy though, and it's probably quite hard to write such components.
It can be a bit mind warping, but it's not bad. I've been doing it with straight ajax in weblog. I'll migrate stuff to pengines when I get free cycles, though I'd rather be making new stuff. Probably first will be a \penginify that works like \ajaxify
>They might be easy to use though. For a Prolog programmers that is, not for the typical web front-end programmer. You can't please them all.
I think I *can* 'please them all' - most web app developers actually work in something like Drupal. We're sitting on a set of fundamentals that would make for a much less rigid but equally 'easy things are easy' version of Drupal. I'm heading towards a system that's easy to use in an idiomatic manner without a lot of understanding of whats underneath. Since whats underneath is the coolness we all know and love, people don't end up being bitten when they stray off the rails.
* mobile support
>That's usually supported by modern JS/CSS frameworks. Responsive design they call it, and in some cases it lets you write once, run anywhere. Or what kind of mobile support are you talking about?
No, I mean a way to write a true mobile app in Prolog, not a browser version.
And, coming along, I expect to see internet of things objects and social robots as interactors soon.
>>Probably [XMPP and/or IRC] best as a pack. Might be a good project for one of my students this fall.
>That would be great!
Definitely.
>>Another widely useful application would be a presence library. This would present a list of people in the room, control muting and banning, handle IP bans, etc. etc.
>A far as I know, XMPP does presence.
Definitely the presence library would have to know about the IRC and XMPP standards.
This could be tied to some authentication/logon solution - atop OpenID etc there's a bunch of other layers - CAS single signon is one I've interacted with, StormPath is a commercial solution. The notion of a user is so pervasive it seems a good thing to have a no-brainer solution for.
>>How's this work with the notion of an application?
>Don't know yet.
I like the idea of an application as a sandbox that anybody can play in, that has certain facilities.
beyond the request/response model, I'd invite considering websockets. Forever saddling us with client-must-initiate is a heavy penalty.
>And is it reasonable for pengine IO to be through websockets?
Yes, that seems reasonable to me - indeed, my interest in pengines was that it seemed to be a way to get websocket like behavior.
>Sure, that's part of the long-term plan. It's probably a lot of work though, and I don't think it's prioritized at the moment, not by me nor by Jan. But one never knows about Jan, he might be sending us a websocket-enabled pengines.pl tomorrow!
that's too bad. The whole objective was to get out of the client-must-initiate box!
What's pengines doing internally? I don't care what the underlying technology is, I just want to nto be forced to do insane things because the server's trying to update the client. (Remember when 'server push' was the buzzword of the moment, back in the 90's?)
> Yep, I think the relation between Prolog and JavaScript is, as I think you can put it, "a marriage made in heaven".
Nope, it's more like a marriage made with daddy's shotgun when Thelma Lou got knocked up. But it's fixable. Them kids might do OK anyway.
Anne Ogborn
That sounds interesting. I'd like to learn LSL.
OK, lets see how many express interest, then we can set up time.
I have written a chatbot, and I'd like to get it to appear
in Second Life in some form.
I don't know if this would be difficult.
No, it's pretty trivial. Making an httprequest is easy.
You just run it as a server.
Jan Wielemaker
> hub is nice. I still think collaboration is better. It's the usual name for such applications.
>
> If we can live with discontiguous (I have to look that spelling up every time I use it) we can
> live with 13 chars in collaboration.
communication, which is precisely
what the library does. Collaboration is enabled by the hub :-)
Compared to an airport, the
library provides the landing strips and the gates, but not the stuff in
between.
>
> 8cD
>
> ::Annie is here, typing in DCG's for a language with keywords like onomatopoeia, apoptosis and prestidigitation.::
in my school carreer ...
Cheers --- Jan
Torbjörn Lager
In this thread, a long time ago, I wrote:
> It should be possible I suppose to allow a pengine to enter into/be
> created in a workspace and for other people (or pengines) in that
> space to "chat" with it (i.e. send it requests and listen for
> seems to provide exactly what is needed for an IDE for collaborative
> more pengines in a workspace equipped with editors, debuggers and
> what have you.
----------
:- module(pengine_chat_client,
[ pengine_enter/2
]).
:- use_module(library(http/websocket)).
:- use_module(library(pengines)).
pengine_enter(URL, Options) :-
atomic_list_concat([URL, '/chat'], WSURL),
http_open_websocket(WSURL, WebSocket, []),
pengine_create(Options),
pengine_event_loop(send_and_receive(WebSocket), []).
send_and_receive(WebSocket, PengineEvent) :-
ws_send(WebSocket, prolog(PengineEvent)),
receive(WebSocket).
receive(WebSocket) :-
( ws_receive(WebSocket, Message, [
format(prolog),
syntax_errors(quiet)
]),
interpret(Message.data)
-> true
; receive(WebSocket)
).
interpret(ID < ask(Goal)) :-
pengine_ask(ID, Goal, []).
interpret(ID < ask(Goal, Options)) :-
pengine_ask(ID, Goal, Options).
interpret(ID < next) :-
pengine_next(ID, []).
interpret(ID < stop) :-
pengine_stop(ID, []).
interpret(ID < abort) :-
pengine_abort(ID, []).
interpret(ID < destroy) :-
pengine_destroy(ID, []).
----------
Given a chat server running a 'http://localhost:3050':
?- use_module(pengine_chat_client).
?- pengine_enter('http://localhost:3050', [alias(foo), destroy(false)]).
Here's what happens in a browser pointing at 'http://localhost:3050'. The lines beginning with "foo < ..." are entered in the clients input field by the user.
create('28fa03aa-cd2f-470c-932a-8d1b14b1ca59',[slave_limit(3)]) .
foo < ask(member(X, [a,b,c])).
success('28fa03aa-cd2f-470c-932a-8d1b14b1ca59',[member(a,[a,b,c])],true) .
foo < next.
success('28fa03aa-cd2f-470c-932a-8d1b14b1ca59',[member(b,[a,b,c])],true) .
foo < stop.
stop('28fa03aa-cd2f-470c-932a-8d1b14b1ca59') .
foo < ask(length([a,b,c], N), [template(N)]).
success('28fa03aa-cd2f-470c-932a-8d1b14b1ca59',[3],false) .
More than one pengine client may enter into the chatroom, and more than one human client can control them and watch their output.
Cheers,
Torbjörn
Jan Wielemaker
[included the mailinglist]
On 08/15/2014 08:35 AM, Anne Ogborn wrote:
>
>
> I've been seeing a 2 second delay in the response when I reload a
> page in the whiteboard.
>
> (the 'network' tab in Chrome's dev tools is useful for this).
>
> ok, thread starving - I set workers to 20 as a test, that clears it
> up. Looking at the network, it's serving 6 things simultaneously -
> diagrammer.css jquiery-2.0.3.min.js diagrammer.js rect.png
> oval.png diamond.png jquery-2.0.3.min.map
Yes, seems that the default 4 workers is insufficient these days. In
the war for faster browsers, IE 11 and Opera 10 go to 8 concurrent
connections. Most others seem to be using 6. Having more than one
was probably a good idea, but I have the impression this is just
trying to get more resources from the server and network than your
neighbor :-(
> some 30ms later it opens the websocket
>
> But now I'm wondering about scaling.
>
> So far I've been noodling around with the core design, trying
> different architectures, but now I'm going to add features.
>
> One of the first things I'll add is the rest of the tools. that means
> adding a batch more icons. This makes me wonder how scalable the
> SWI-Prolog server actually is when you have a lot of small assets and
> multiple users doing interleaved requests? This could be especially
> problematic if you have several large assets - imagine a web page
> with lots of tiny images and a few large ones, where you have
> unfavorable blocking.
SWI-Prolog's design is basically a one-thread-per-connection design.
This is (AFAIK) also the case for Apache, except that they distribute
the threads over multiple processes. We could do the same (everything
is there to do that on *nix systems), but it looses the nice property
that you have a shared dynamic database. Some servers, like nginx,
use I/O multiplexing. That is cheaper. We probably could do that
as well for tasks such as serving static files. E.g., http_reply_file
could hand the connection to a dedicated (set of) thread(s) using
I/O multiplexing to serve the files.
www.swi-prolog.org using the thread-pooling options to separate tasks
with diverse properties over different pools. So, short files and
simple requests are handled by the 16 default workers, while
long-running download tasks are spawned on their own thread. So are
autocompletion requests, mainly trying to avoid them from flooding the
server. Ok, the site has only about 4,000 visiters doing a bit over
100,000 requests per day and serves just 300Gb per month. It
sometimes suffers from poor response. As far as I managed to
diagnose, these are (mostly) caused by network overload due to
maintenance tasks in the university IT infrastructure.
Another option is to reduce the number of accepted keep-alive
connections (possible per client IP) when approaching limits.
On the client side you can do a lot by reducing the number of files
(combine JS files, combine CSS, embed icons in CSS, etc.).
For short, I wouldn't worry too much. When applications arrive that
require more scalability, there is nothing in the design that stops
us improving the server's resource handling.
Cheers --- Jan
Michael Hendricks
Yes, seems that the default 4 workers is insufficient these days. In
the war for faster browsers, IE 11 and Opera 10 go to 8 concurrent
connections. Most others seem to be using 6. Having more than one
was probably a good idea, but I have the impression this is just
trying to get more resources from the server and network than your
neighbor :-(
SWI-Prolog's design is basically a one-thread-per-connection design.
This is (AFAIK) also the case for Apache, except that they distribute
the threads over multiple processes. We could do the same (everything
is there to do that on *nix systems), but it looses the nice property
that you have a shared dynamic database.
Some servers, like nginx,
use I/O multiplexing. That is cheaper. We probably could do that
as well for tasks such as serving static files. E.g., http_reply_file
could hand the connection to a dedicated (set of) thread(s) using
I/O multiplexing to serve the files.
www.swi-prolog.org using the thread-pooling options to separate tasks
with diverse properties over different pools. So, short files and
simple requests are handled by the 16 default workers, while
long-running download tasks are spawned on their own thread. So are
autocompletion requests, mainly trying to avoid them from flooding the
server. Ok, the site has only about 4,000 visiters doing a bit over
100,000 requests per day and serves just 300Gb per month. It
sometimes suffers from poor response. As far as I managed to
diagnose, these are (mostly) caused by network overload due to
maintenance tasks in the university IT infrastructure.
Another option is to reduce the number of accepted keep-alive
connections (possible per client IP) when approaching limits.
On the client side you can do a lot by reducing the number of files
(combine JS files, combine CSS, embed icons in CSS, etc.).
For short, I wouldn't worry too much.
Jan Wielemaker
Thanks for the insight that we can fix most of these issues externally.
I didn't know that nginx could be used to avoid that clients behind slow
networks keep too many connections open. Serving static files using
alternative mechanisms is of course a good thing for production. It is
nice that Prolog can do that such that small installations and testing
does not require an additional webserver running side-by-side.
Once you go production, just put an nginx proxy in front of Prolog and
make nginx serve the directories holding static files directly.
On 08/15/2014 04:24 PM, Michael Hendricks wrote:
> When running SWI Prolog in these contexts, the current
> thread-per-connection approach works pretty well. That probably changes
> for websockets where SWI handles more of the traffic itself.
standard library) should handle many websockets quite elegantly. It
bundles N websockets per thread, using select() to wait for input.
Currently N is 64.
That said, I have little clue how many threads the HTTP server can run
and where the bottlenecks will appear. It probably depends enourmously
on what the threads do, the OS and the hardware. Would be nice to have
some figures.
Cheers --- Jan