I'm trying to understand a detail about net.createServer().
In the code below, how is it that Node guarantees that no data will be lost
if a client immediately writes to its socket when it connects to this
server?
Is something buffering writes from the client until the callback function
passed to net.createServer completes?
Does the server not read from its socket until that callback completes
because something that enables reads is waiting on the event queue?
var net = require('net');
var server = net.createServer(function (socket) {
// Could the client write to the socket before the next call completes?
socket.on('data', function (data) {
console.log('received "' + data + '"');
});
});
server.listen(8123, function () {
console.log('listening');
> I'm trying to understand a detail about net.createServer().
> In the code below, how is it that Node guarantees that no data will be lost if a client immediately writes to its socket when it connects to this server?
> Is something buffering writes from the client until the callback function passed to net.createServer completes?
> Does the server not read from its socket until that callback completes because something that enables reads is waiting on the event queue?
> var net = require('net');
> var server = net.createServer(function (socket) {
> // Could the client write to the socket before the next call completes?
> socket.on('data', function (data) {
> console.log('received "' + data + '"');
> });
> });
> server.listen(8123, function () {
> console.log('listening');
> });
My understanding is that node won't/shouldn't emit/dispatch any 'data' events to the socket 'socket' until *after* having called cb(socket), 'cb' being the callback function passed on to .createServer(cb).
-- Jorge.
On Thu, Sep 6, 2012 at 4:40 PM, Jorge <jo...@jorgechamorro.com> wrote:
> On 06/09/2012, at 23:11, Mark Volkmann wrote:
> My understanding is that node won't/shouldn't emit/dispatch any 'data'
> events to the socket 'socket' until *after* having called cb(socket), 'cb'
> being the callback function passed on to .createServer(cb).
I suspect you are correct. I'd like to understand how Node implements that.
It would be great if this is already documented somewhere and I could just
read that to understand it.
> On Thu, Sep 6, 2012 at 4:40 PM, Jorge <jo...@jorgechamorro.com> wrote:
>> On 06/09/2012, at 23:11, Mark Volkmann wrote:
>> My understanding is that node won't/shouldn't emit/dispatch any 'data'
>> events to the socket 'socket' until *after* having called cb(socket), 'cb'
>> being the callback function passed on to .createServer(cb).
> I suspect you are correct. I'd like to understand how Node implements
> that. It would be great if this is already documented somewhere and I could
> just read that to understand it.
-- AUFKLÄRUNG ist der Ausgang des Menschen aus seiner selbstverschuldeten
Unmündigkeit. Unmündigkeit ist das Unvermögen, sich seines Verstandes ohne
Leitung eines anderen zu bedienen. Selbstverschuldet ist diese
Unmündigkeit, wenn die Ursache derselben nicht am Mangel des Verstandes,
sondern der Entschließung und des Mutes liegt, sich seiner ohne Leitung
eines andern zu bedienen. Sapere aude! Habe Mut, dich deines eigenen
Verstandes zu bedienen! ist also der Wahlspruch der Aufklärung.
> It is a good way to understand through source code.
> 2012/9/7 Mark Volkmann <r.mark.volkm...@gmail.com>
>> On Thu, Sep 6, 2012 at 4:40 PM, Jorge <jo...@jorgechamorro.com> wrote:
>>> On 06/09/2012, at 23:11, Mark Volkmann wrote:
>>> My understanding is that node won't/shouldn't emit/dispatch any 'data'
>>> events to the socket 'socket' until *after* having called cb(socket), 'cb'
>>> being the callback function passed on to .createServer(cb).
>> I suspect you are correct. I'd like to understand how Node implements
>> that. It would be great if this is already documented somewhere and I could
>> just read that to understand it.
>> --
>> R. Mark Volkmann
>> Object Computing, Inc.
> --
> AUFKLÄRUNG ist der Ausgang des Menschen aus seiner selbstverschuldeten
> Unmündigkeit. Unmündigkeit ist das Unvermögen, sich seines Verstandes ohne
> Leitung eines anderen zu bedienen. Selbstverschuldet ist diese
> Unmündigkeit, wenn die Ursache derselben nicht am Mangel des Verstandes,
> sondern der Entschließung und des Mutes liegt, sich seiner ohne Leitung
> eines andern zu bedienen. Sapere aude! Habe Mut, dich deines eigenen
> Verstandes zu bedienen! ist also der Wahlspruch der Aufklärung.
I don't doubt you. I just want to see that documented somewhere or clearly
see how that is enforced in the code. I have looked at net.js. It's not the
easiest code to follow. If it's clear from that code, I could use some
pointers on where to look.
I'll try to make my concern more clear below with very simple code for
server.js and client.js that really runs. Run "node server" in one window
and "node client" in another.
The numbered comments indicate the order in which I expect the lines to
execute.
It seems clear to me that 4 could run before 5. If that happens, how is it
that the server still gets that message? Is the message being buffered
until connListener completes?
server.js
var net = require('net');
function dataListener(data) {
console.log('received', data.toString());
}
function connListener(socket) {
socket.on('data', dataListener); // 5
}
var server = net.createServer(connListener); // 1
server.listen(8019); // 2
client.js
var net = require('net');
var socket = net.connect(8019); // 3
socket.write('Is this lost?'); // 4
>> It is a good way to understand through source code.
>> 2012/9/7 Mark Volkmann <r.mark.volkm...@gmail.com>
>>> On Thu, Sep 6, 2012 at 4:40 PM, Jorge <jo...@jorgechamorro.com> wrote:
>>>> On 06/09/2012, at 23:11, Mark Volkmann wrote:
>>>> My understanding is that node won't/shouldn't emit/dispatch any 'data'
>>>> events to the socket 'socket' until *after* having called cb(socket), 'cb'
>>>> being the callback function passed on to .createServer(cb).
>>> I suspect you are correct. I'd like to understand how Node implements
>>> that. It would be great if this is already documented somewhere and I could
>>> just read that to understand it.
On Fri, Sep 7, 2012 at 3:35 PM, Mark Volkmann <r.mark.volkm...@gmail.com> wrote:
> I don't doubt you. I just want to see that documented somewhere or clearly
> see how that is enforced in the code. I have looked at net.js. It's not the
> easiest code to follow. If it's clear from that code, I could use some
> pointers on where to look.
> I'll try to make my concern more clear below with very simple code for
> server.js and client.js that really runs. Run "node server" in one window
> and "node client" in another.
> The numbered comments indicate the order in which I expect the lines to
> execute.
> It seems clear to me that 4 could run before 5. If that happens, how is it
> that the server still gets that message? Is the message being buffered until
> connListener completes?
> server.js
> var net = require('net');
> function dataListener(data) {
> console.log('received', data.toString());
> }
> function connListener(socket) {
> socket.on('data', dataListener); // 5
> }
> var server = net.createServer(connListener); // 1
> server.listen(8019); // 2
> client.js
> var net = require('net');
> var socket = net.connect(8019); // 3
> socket.write('Is this lost?'); // 4
The client queues write requests until the connection handshake is
done. Said handshake is complete when the server accepts the
connection, which node does right before it calls your connListener
function.
> I don't doubt you. I just want to see that documented somewhere or clearly
> see how that is enforced in the code. I have looked at net.js. It's not the
> easiest code to follow. If it's clear from that code, I could use some
> pointers on where to look.
> I'll try to make my concern more clear below with very simple code for
> server.js and client.js that really runs. Run "node server" in one window
> and "node client" in another.
> The numbered comments indicate the order in which I expect the lines to
> execute.
> It seems clear to me that 4 could run before 5. If that happens, how is it
> that the server still gets that message? Is the message being buffered
> until connListener completes?
> server.js
> var net = require('net');
> function dataListener(data) {
> console.log('received', data.toString());
> }
> function connListener(socket) {
> socket.on('data', dataListener); // 5
> }
> var server = net.createServer(connListener); // 1
> server.listen(8019); // 2
> client.js
> var net = require('net');
> var socket = net.connect(8019); // 3
> socket.write('Is this lost?'); // 4
> On Fri, Sep 7, 2012 at 7:29 AM, ribao wei <riba...@gmail.com> wrote:
>> It is not about node.js, it is a Javascript thing.
>> Socket will not receive any data util it "connect".
>> On Thu, Sep 6, 2012 at 11:35 PM, Hsu Ping Feng <fillano.f...@gmail.com>wrote:
>>> It is a good way to understand through source code.
>>> 2012/9/7 Mark Volkmann <r.mark.volkm...@gmail.com>
>>>> On Thu, Sep 6, 2012 at 4:40 PM, Jorge <jo...@jorgechamorro.com> wrote:
>>>>> On 06/09/2012, at 23:11, Mark Volkmann wrote:
>>>>> My understanding is that node won't/shouldn't emit/dispatch any 'data'
>>>>> events to the socket 'socket' until *after* having called cb(socket), 'cb'
>>>>> being the callback function passed on to .createServer(cb).
>>>> I suspect you are correct. I'd like to understand how Node implements
>>>> that. It would be great if this is already documented somewhere and I could
>>>> just read that to understand it.
On Fri, Sep 7, 2012 at 9:09 AM, Ben Noordhuis <i...@bnoordhuis.nl> wrote:
The client queues write requests until the connection handshake is
> done. Said handshake is complete when the server accepts the
> connection, which node does right before it calls your connListener
> function.
If that is true then why is the first message from the client lost if I do
this?
Note that on my machine (OSX) the message is lost if the timeout is 2 ms or
more, but not if it is less than 2.
Maybe the client queues write requests until the connListener function
completes.
server.js
var net = require('net');
function dataListener(data) {
console.log('received', data.toString());
}
function connListener(socket) {
setTimeout(function () {
socket.on('data', dataListener);
}, 2);
}
var server = net.createServer(connListener);
server.listen(8019);
client.js
var net = require('net');
var socket = net.connect(8019);
socket.write('Is this lost?');
> On Fri, Sep 7, 2012 at 9:09 AM, Ben Noordhuis <i...@bnoordhuis.nl> wrote:
> The client queues write requests until the connection handshake is
>> done. Said handshake is complete when the server accepts the
>> connection, which node does right before it calls your connListener
>> function.
> If that is true then why is the first message from the client lost if I do
> this?
> Note that on my machine (OSX) the message is lost if the timeout is 2 ms
> or more, but not if it is less than 2.
> Maybe the client queues write requests until the connListener function
> completes.
> server.js
> var net = require('net');
> function dataListener(data) {
> console.log('received', data.toString());
> }
> function connListener(socket) {
> setTimeout(function () {
> socket.on('data', dataListener);
> }, 2);
> }
> var server = net.createServer(connListener);
> server.listen(8019);
> client.js
> var net = require('net');
> var socket = net.connect(8019);
> socket.write('Is this lost?');
If the server accepts the connection BEFORE connListener is called, then it
seems possible that the client could send a message before the server
registers for data events. That's why it seems that the client queues
messages until AFTER connListener completes. If I'm wrong about this, I'd
like to understand why.
On Fri, Sep 7, 2012 at 9:25 AM, ribao wei <riba...@gmail.com> wrote:
> Well, in this code connListener finishes execution without dataListener
> being registered. So I guess that is why the first message is lost.
> On Fri, Sep 7, 2012 at 10:21 AM, Mark Volkmann <r.mark.volkm...@gmail.com>wrote:
>> On Fri, Sep 7, 2012 at 9:09 AM, Ben Noordhuis <i...@bnoordhuis.nl> wrote:
>> The client queues write requests until the connection handshake is
>>> done. Said handshake is complete when the server accepts the
>>> connection, which node does right before it calls your connListener
>>> function.
>> If that is true then why is the first message from the client lost if I
>> do this?
>> Note that on my machine (OSX) the message is lost if the timeout is 2 ms
>> or more, but not if it is less than 2.
>> Maybe the client queues write requests until the connListener function
>> completes.
>> server.js
>> var net = require('net');
>> function dataListener(data) {
>> console.log('received', data.toString());
>> }
>> function connListener(socket) {
>> setTimeout(function () {
>> socket.on('data', dataListener);
>> }, 2);
>> }
>> var server = net.createServer(connListener);
>> server.listen(8019);
>> client.js
>> var net = require('net');
>> var socket = net.connect(8019);
>> socket.write('Is this lost?');
>> --
>> R. Mark Volkmann
>> Object Computing, Inc.
On Sep 7, 10:21 am, Mark Volkmann <r.mark.volkm...@gmail.com> wrote:
> On Fri, Sep 7, 2012 at 9:09 AM, Ben Noordhuis <i...@bnoordhuis.nl> wrote:
> The client queues write requests until the connection handshake is
> > done. Said handshake is complete when the server accepts the
> > connection, which node does right before it calls your connListener
> > function.
> If that is true then why is the first message from the client lost if I do
> this?
Probably because the accept and incoming data happen within the same
tick. This is why you should add a data handler right away, or at
least pause() the socket right away.
On Fri, Sep 7, 2012 at 12:01 PM, mscdex <msc...@gmail.com> wrote:
> On Sep 7, 10:21 am, Mark Volkmann <r.mark.volkm...@gmail.com> wrote:
> > On Fri, Sep 7, 2012 at 9:09 AM, Ben Noordhuis <i...@bnoordhuis.nl>
> wrote:
> > The client queues write requests until the connection handshake is
> > > done. Said handshake is complete when the server accepts the
> > > connection, which node does right before it calls your connListener
> > > function.
> > If that is true then why is the first message from the client lost if I
> do
> > this?
> Probably because the accept and incoming data happen within the same
> tick. This is why you should add a data handler right away, or at
> least pause() the socket right away.
Sounds reasonable. I'm currently teaching a class on Node.js. I'd like to
say something better to the students than "This code probably works because
...". That's why I'm trying to either find a place where this is documented
or find where the queuing of messages and releasing of them actually
happens in the code.
It sounds like you might be misunderstanding something fundamental about node/Javascript - it's all single threaded, and (at least the networking stuff) is event-based (specifically a stream of sequential events, since there's no concurrency). In your example, while it's running line 5, there's no way it's going to miss anything because that's the only code that is running, nothing could possibly be pumping the event loop. Until your function returns from handling the event, node is not going to handle the next event coming from the socket. Even if the data has been sent and buffered in the kernel for your server process, it's not going to fire any events until node tries to read from it/pumps the event loop. Lots of node APIs rely on this kind of behavior (e.g. HTTP request/response stuff similarly lets you set up event listeners in a connection event before it could possibly fire any data events). Hope this helps!
On Friday, September 7, 2012 10:35:28 AM UTC-7, Mark Volkmann wrote:
> On Fri, Sep 7, 2012 at 12:01 PM, mscdex <msc...@gmail.com <javascript:>>wrote:
>> On Sep 7, 10:21 am, Mark Volkmann <r.mark.volkm...@gmail.com> wrote: >> > On Fri, Sep 7, 2012 at 9:09 AM, Ben Noordhuis <i...@bnoordhuis.nl> >> wrote:
>> > The client queues write requests until the connection handshake is
>> > > done. Said handshake is complete when the server accepts the >> > > connection, which node does right before it calls your connListener >> > > function.
>> > If that is true then why is the first message from the client lost if I >> do >> > this?
>> Probably because the accept and incoming data happen within the same >> tick. This is why you should add a data handler right away, or at >> least pause() the socket right away.
> Sounds reasonable. I'm currently teaching a class on Node.js. I'd like to > say something better to the students than "This code probably works because > ...". That's why I'm trying to either find a place where this is documented > or find where the queuing of messages and releasing of them actually > happens in the code.
On Fri, Sep 7, 2012 at 1:16 PM, Jimb Esser <wastel...@gmail.com> wrote:
> It sounds like you might be misunderstanding something fundamental about
> node/Javascript - it's all single threaded, and (at least the networking
> stuff) is event-based (specifically a stream of sequential events, since
> there's no concurrency). In your example, while it's running line 5,
> there's no way it's going to miss anything because that's the only code
> that is running, nothing could possibly be pumping the event loop. Until
> your function returns from handling the event, node is not going to handle
> the next event coming from the socket. Even if the data has been sent and
> buffered in the kernel for your server process, it's not going to fire any
> events until node tries to read from it/pumps the event loop. Lots of node
> APIs rely on this kind of behavior (e.g. HTTP request/response stuff
> similarly lets you set up event listeners in a connection event before it
> could possibly fire any data events). Hope this helps!
What I think you're saying is that the server will stop queuing messages
AND execute the connection listener in the same synchronous tick. So
there's no way a queued message will be delivered until after the
connection listener runs. That makes sense.
BTW, if I didn't understand that Node is single-threaded, I really
shouldn't be attempting to teach a class on it. I do understand that part.
;-)
An off-the-shelf node can't execute *JavaScript* code in parallel (*), because *only* *one* of its *multiple* threads runs a (single) v8 JS VM.
But a node with the threads-a-gogo module can create as many (up to thousands) additional V8 JS VMs as needed, and execute javascript code truly in parallel, in additional threads:
> It sounds like you might be misunderstanding something fundamental about node/Javascript - it's all single threaded, and (at least the networking stuff) is event-based (specifically a stream of sequential events, since there's no concurrency). In your example, while it's running line 5, there's no way it's going to miss anything because that's the only code that is running, nothing could possibly be pumping the event loop. Until your function returns from handling the event, node is not going to handle the next event coming from the socket. Even if the data has been sent and buffered in the kernel for your server process, it's not going to fire any events until node tries to read from it/pumps the event loop. Lots of node APIs rely on this kind of behavior (e.g. HTTP request/response stuff similarly lets you set up event listeners in a connection event before it could possibly fire any data events). Hope this helps!
> Jimb Esser
> On Friday, September 7, 2012 10:35:28 AM UTC-7, Mark Volkmann wrote:
> On Fri, Sep 7, 2012 at 12:01 PM, mscdex <msc...@gmail.com> wrote:
> On Sep 7, 10:21 am, Mark Volkmann <r.mark.volkm...@gmail.com> wrote:
> > On Fri, Sep 7, 2012 at 9:09 AM, Ben Noordhuis <i...@bnoordhuis.nl> wrote:
> > The client queues write requests until the connection handshake is
> > > done. Said handshake is complete when the server accepts the
> > > connection, which node does right before it calls your connListener
> > > function.
> > If that is true then why is the first message from the client lost if I do
> > this?
> Probably because the accept and incoming data happen within the same
> tick. This is why you should add a data handler right away, or at
> least pause() the socket right away.
> Sounds reasonable. I'm currently teaching a class on Node.js. I'd like to say something better to the students than "This code probably works because ...". That's why I'm trying to either find a place where this is documented or find where the queuing of messages and releasing of them actually happens in the code.
On Fri, Sep 7, 2012 at 7:05 PM, Mark Volkmann <r.mark.volkm...@gmail.com>wrote:
> I don't doubt you. I just want to see that documented somewhere or clearly
> see how that is enforced in the code. I have looked at net.js. It's not the
> easiest code to follow. If it's clear from that code, I could use some
> pointers on where to look.
> I'll try to make my concern more clear below with very simple code for
> server.js and client.js that really runs. Run "node server" in one window
> and "node client" in another.
> The numbered comments indicate the order in which I expect the lines to
> execute.
> It seems clear to me that 4 could run before 5. If that happens, how is it
> that the server still gets that message? Is the message being buffered
> until connListener completes?
4 always runs before 5. The data event cannot be triggered unless something
is written to the socket. This is provided the server.js file is run first
and then client.js. The other way round will not work as it will throw a
ECONNREFUSED on connect() at line 3 (provided the server has not done a
listen() yet).
Here is the flow:
Behind the scenes, the event loop is notified of a new connection. It
accept()'s the connection. Once accept()ed the connListener callback is
triggered and the particular socket (wrapper to the raw socket) is passed
as an argument. Listeners for various events (such as "data", "end",
"chunk" etc) are setup. Then, a read event handler is registered on the raw
socket's FD (UV_READ). When the file descriptor becomes readable, the data
is recv()d and is stored in a buffer and "chunk" event is emitted for each
data received (recv() returns the size of the data and "errno" is EAGAIN
until all data is read into the buffer). When the buffer is full (recv()
returns a -1), the "data" event is emitted with the buffer as argument. If
recv() returns 0, the socket is deemed to have closed on the other end. The
read event handler is then stopped, a full shutdown of the raw socket is
performed and the file descriptor is detached from the event loop with the
"end" event being emitted. (i have simplified it a lot. this is how its
generally done).
>>> It is a good way to understand through source code.
>>> 2012/9/7 Mark Volkmann <r.mark.volkm...@gmail.com>
>>>> On Thu, Sep 6, 2012 at 4:40 PM, Jorge <jo...@jorgechamorro.com> wrote:
>>>>> On 06/09/2012, at 23:11, Mark Volkmann wrote:
>>>>> My understanding is that node won't/shouldn't emit/dispatch any 'data'
>>>>> events to the socket 'socket' until *after* having called cb(socket), 'cb'
>>>>> being the callback function passed on to .createServer(cb).
>>>> I suspect you are correct. I'd like to understand how Node implements
>>>> that. It would be great if this is already documented somewhere and I could
>>>> just read that to understand it.
> On Fri, Sep 7, 2012 at 7:05 PM, Mark Volkmann <r.mark.volkm...@gmail.com>wrote:
>> I don't doubt you. I just want to see that documented somewhere or
>> clearly see how that is enforced in the code. I have looked at net.js. It's
>> not the easiest code to follow. If it's clear from that code, I could use
>> some pointers on where to look.
>> I'll try to make my concern more clear below with very simple code for
>> server.js and client.js that really runs. Run "node server" in one window
>> and "node client" in another.
>> The numbered comments indicate the order in which I expect the lines to
>> execute.
>> It seems clear to me that 4 could run before 5. If that happens, how is
>> it that the server still gets that message? Is the message being buffered
>> until connListener completes?
> 4 always runs before 5. The data event cannot be triggered unless
> something is written to the socket. This is provided the server.js file is
> run first and then client.js. The other way round will not work as it will
> throw a ECONNREFUSED on connect() at line 3 (provided the server has not
> done a listen() yet).
> Here is the flow:
> Behind the scenes, the event loop is notified of a new connection. It
> accept()'s the connection. Once accept()ed the connListener callback is
Oops. Should be, the server is notified of a new connection via the event
loop. The server accept()'s the connection....
> triggered and the particular socket (wrapper to the raw socket) is passed
> as an argument. Listeners for various events (such as "data", "end",
> "chunk" etc) are setup. Then, a read event handler is registered on the raw
> socket's FD (UV_READ). When the file descriptor becomes readable, the data
> is recv()d and is stored in a buffer and "chunk" event is emitted for each
> data received (recv() returns the size of the data and "errno" is EAGAIN
> until all data is read into the buffer). When the buffer is full (recv()
> returns a -1), the "data" event is emitted with the buffer as argument. If
> recv() returns 0, the socket is deemed to have closed on the other end. The
> read event handler is then stopped, a full shutdown of the raw socket is
> performed and the file descriptor is detached from the event loop with the
> "end" event being emitted. (i have simplified it a lot. this is how its
> generally done).
>> server.js
>> var net = require('net');
>> function dataListener(data) {
>> console.log('received', data.toString());
>> }
>> function connListener(socket) {
>> socket.on('data', dataListener); // 5
>> }
>> var server = net.createServer(connListener); // 1
>> server.listen(8019); // 2
>> client.js
>> var net = require('net');
>> var socket = net.connect(8019); // 3
>> socket.write('Is this lost?'); // 4
>> On Fri, Sep 7, 2012 at 7:29 AM, ribao wei <riba...@gmail.com> wrote:
>>> It is not about node.js, it is a Javascript thing.
>>> Socket will not receive any data util it "connect".
>>> On Thu, Sep 6, 2012 at 11:35 PM, Hsu Ping Feng <fillano.f...@gmail.com>wrote:
>>>> It is a good way to understand through source code.
>>>> 2012/9/7 Mark Volkmann <r.mark.volkm...@gmail.com>
>>>>> On Thu, Sep 6, 2012 at 4:40 PM, Jorge <jo...@jorgechamorro.com> wrote:
>>>>>> On 06/09/2012, at 23:11, Mark Volkmann wrote:
>>>>>> My understanding is that node won't/shouldn't emit/dispatch any
>>>>>> 'data' events to the socket 'socket' until *after* having called
>>>>>> cb(socket), 'cb' being the callback function passed on to .createServer(cb).
>>>>> I suspect you are correct. I'd like to understand how Node implements
>>>>> that. It would be great if this is already documented somewhere and I could
>>>>> just read that to understand it.
>> --
>> R. Mark Volkmann
>> Object Computing, Inc.
