Re: [snap] How can I make response chunks send immediately?

Gregory Collins

unread,

Apr 9, 2013, 4:20:36 PM4/9/13

to snap_fr...@googlegroups.com

The "writeBS" function doesn't immediately send the data -- instead the data is queued and sent after the user handler runs. You would need to do your data fetch in your output iteratee. See https://github.com/snapframework/cufp2011 for an example of the http long polling technique.

On Tue, Apr 9, 2013 at 8:02 PM, Neuman Vong <neuma...@gmail.com> wrote:

Hi Snap devs,

I have some code that returns a response, the first part of which is static and the second part dynamic, something like:

site :: Snap ()
site = do
writeBS htmlHead

content <- getSlowContent
writeBS content

When I telnet to the server and issue a request, the response shows "Transfer-Encoding: chunked", and the responses are chunked. However, I expected htmlHead (the static portion) to be written immediately, and then after a delay the content (the dynamic portion). Instead, there's a long delay, and both chunks appear at the same time.

I've tried modifyResponse $ setBuffering False, and also writeBuilder flush.

Am I doing something wrong? There are several mentions of "Comet" in the Snap source, does anyone have an example?

Thanks!

--

---
You received this message because you are subscribed to the Google Groups "Snap Framework" group.
To unsubscribe from this group and stop receiving emails from it, send an email to snap_framewor...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Gregory Collins <gr...@gregorycollins.net>

Neuman Vong

unread,

Apr 10, 2013, 1:09:01 PM4/10/13

to snap_fr...@googlegroups.com

Hi Greg,

Thanks for your response. I asked this question on #snapframwork and mightbyte suggested I look at that project.

I looked at cufp2011 but it seemed like the long polling technique was still based on one request to one response (even though it may be in multiple chunks). I'm looking for something like a server push with one request, for e.g.:

Request:

GET / HTTP/1.1

Host: localhost:8000

Response:

HTTP/1.1 200 OK

Transfer-Encoding: chunked

3

foo

[...an arbitrary delay...]

3

foo

I tried creating a custom enumerator: http://hpaste.org/85513. This seems to work if I omit the threadDelay, otherwise I don't receive a response from the server. I'd appreciate if you could take a look, I have a feeling something like this would be obvious to those familiar with Snap.

The real world context for this is that I want to send the first part of the HTML content the <head>, which is mostly static and available immediately, and then have the dynamic content, which is vulnerable to latency in the backend, sent later.

Gregory Collins

unread,

Apr 10, 2013, 1:28:24 PM4/10/13

to snap_fr...@googlegroups.com

Hi,

Quick sanity check: are you building your executable with "-threaded"?

But it looks like the issue is the lack of builder flush here. Snap buffers your enumerator's output with Builder for efficiency. Change line 23 to

k' <- runIteratee (k (Chunks [bs `mappend` flush]))

and it ought to work.

G

Neuman Vong

unread,

Apr 10, 2013, 2:55:16 PM4/10/13

to snap_fr...@googlegroups.com

Hi Greg,

Not building with "-threaded"? Should I be?

I'll try this change. Thanks.

One last thing, is there a better way to do this? Could I be leveraging Snap's higher level APIs better? Something like:

site = do

write myStaticContent // Sends chunk immediately

dynamicContent <- getDynamicContent // Could take a long time

write dynamicContent

Gregory Collins

unread,

Apr 10, 2013, 3:10:10 PM4/10/13

to snap_fr...@googlegroups.com

Here "site" runs in the Snap monad. No data is sent to the user at all until the Snap monad computation finishes! The reason being, we don't know what the headers will look like until the user handler completely finishes. Your "site" function will probably be wrapped with a compression combinator which will add its own headers (and modify your output enumerator), for instance.

So, in order to get the behaviour you want, you would need to do it directly within the output enumerator. This will get a lot easier in the next major snap version --- I'm still several months away from finishing that however.

Neuman Vong

unread,

Apr 10, 2013, 7:53:33 PM4/10/13

to snap_fr...@googlegroups.com

Ah, thanks.

I'm creating some helper enumBuilders to get an API that looks similar to write* that I can run in the Snap monad, so maybe I'll get it looking simpler hopefully.

I'd be quite interested in seeing the design you have in mind for the next major version. Is this something you're willing to share?

Andrew Cowie

unread,

Apr 10, 2013, 8:30:39 PM4/10/13

to snap_fr...@googlegroups.com

On Wed, 2013-04-10 at 16:53 -0700, Neuman Vong wrote:

> I'd be quite interested in seeing the design you have in mind for the
> next major version. Is this something you're willing to share?

He means the new snap-server is built on io-streams, which was itself
designed with problems like this in mind.

AfC
Sydney

>

Neuman Vong

unread,

Apr 10, 2013, 8:40:05 PM4/10/13

to snap_fr...@googlegroups.com

Hi Andrew,

I think I understand that io-streams are being considered to replace Iteratee IO, is that correct? What I'm trying to understand is if Iteratee IO is able to stream (and I think it does, since we can do this via custom enumerators), then is there a nice looking way of packaging up static/dynamic chunking so it can be used without having to make a custom enumerator.

I imagine that when io-streams arrives, static/dynamic chunking will still be precluded by the higher level Snap API. Is this correct?

Gregory Collins

unread,

Apr 11, 2013, 3:34:12 AM4/11/13

to snap_fr...@googlegroups.com

On Thu, Apr 11, 2013 at 2:40 AM, Neuman Vong <neuma...@gmail.com> wrote:

I imagine that when io-streams arrives, static/dynamic chunking will still be precluded by the higher level Snap API.

I have no idea what you mean by this, sorry. The central idea here is: the output body is a procedure, ultimately running in the IO monad, that will generate binary output when run later. A combinator like "writeBS" that runs in the Snap monad actually just tacks "(>==> (enumBS "foo")) onto that procedure. If I do this:

foo = do

writeBS "A"

modifyResponse $ setResponseBody (enumBS "B")

Then "B" gets written as the response body. The reason the response body runs on top of IO (or here "Iteratee IO") and not the Snap monad is because by the time the response is being sent, all decisions have been made: we've written the response line and headers to the output. The Snap monad is just a little bit of state (Request, Response, and a couple of other things) over "Iteratee IO" anyways.

Your goal in most code should be to get as much *out* of the Snap monad as possible: best being pure referentially transparent functions, and IO when otherwise possible. Or, let's say you use "(<|>)" but nothing else -- that function can get an "(Alternative m, Monad m) => ..." constraint and work with Parser and logic backtracking monads too. Code that runs in the Snap monad is usable only within a webserver; code that runs in IO works everywhere. So usually in my own projects I like to see only the code that deals with HTTP living in the Snap monad.

G

--
Gregory Collins <gr...@gregorycollins.net>

MightyByte

unread,

Apr 11, 2013, 10:27:13 AM4/11/13

to snap_fr...@googlegroups.com

FWIW, here's my take on interpreting your questions. (Greg can correct me if I'm wrong about his plans.)

On Wed, Apr 10, 2013 at 8:40 PM, Neuman Vong <neuma...@gmail.com> wrote:

I think I understand that io-streams are being considered to replace Iteratee IO, is that correct? What I'm trying to understand is if Iteratee IO is able to stream (and I think it does, since we can do this via custom enumerators), then is there a nice looking way of packaging up static/dynamic chunking so it can be used without having to make a custom enumerator.

It's not just being considered. I believe it's pretty much guaranteed that we will completely replace the use of the enumerator package (which includes iteratees) with io-streams. At this point it's just a matter of getting it implemented and tested.

I imagine that when io-streams arrives, static/dynamic chunking will still be precluded by the higher level Snap API. Is this correct?

The higher level Snap monad API will stay the same except to change any use of iteratees/enemurators to use the io-streams equivalent. The concepts don't change. You will still have to drop down to the io-stream level to do the delay that you've been asking about.

Neuman Vong

unread,

Apr 14, 2013, 2:25:01 PM4/14/13

to snap_fr...@googlegroups.com

No, it's most probably because I phrased the question poorly. I'm not yet familiar enough with Snap terminology to be precise.

I see now about the Snap monad only being state over Iteratee IO. I was most interested to know if the Snap monad would change in the future to also handle cases where sending a response chunk immediately requires using Iteratee IO directly, and if so, how would that look.

I think mightybyte may correctly guessed what I was looking for despite my vaguely posed question.

Neuman Vong

unread,

Apr 14, 2013, 2:25:59 PM4/14/13

to snap_fr...@googlegroups.com

Hi,

On Thursday, April 11, 2013 7:27:13 AM UTC-7, mightybyte wrote:

The higher level Snap monad API will stay the same except to change any use of iteratees/enemurators to use the io-streams equivalent. The concepts don't change. You will still have to drop down to the io-stream level to do the delay that you've been asking about.

Yes, this is what I wanted to know. Thank you!

Reply all

Reply to author

Forward