[AOLSERVER] Does not work ns_return + zlib

32 views
Skip to first unread message

Alexey Pechnikov

unread,
Nov 30, 2010, 7:31:48 PM11/30/10
to AOLS...@listserv.aol.com
The code below does not produce valid output in AOL 4.5.1. But why?..

ns_register_proc GET  /headers  ad_headers_proc

proc ad_headers_proc {ignore} {
    set result {}
    set headers [ns_conn headers]
    for {set i 0} {$i < [ns_set size $headers]} {incr i} {
        set key [ns_set key $headers $i]
        set value [ns_set value $headers $i]
        append result "$key: $value\n"
    }
    ns_updateheader Content-Encoding gzip
    ns_return 200 text/plain [ns_zlib gzip $result]
}


--
Best regards, Alexey Pechnikov.
http://pechnikov.tel/

-- AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to <list...@listserv.aol.com> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.

russell muetzelfeldt

unread,
Nov 30, 2010, 8:01:05 PM11/30/10
to AOLS...@listserv.aol.com
On 01/12/2010, at 11:31 AM, Alexey Pechnikov wrote:

> The code below does not produce valid output in AOL 4.5.1. But why?..

does it produce valid output in other versions of nsd?

The nsreturnz sources seem to be saying that a response with the gzip Content-Encoding isn't just a plain gzip compressed copy of the uncompressed body -

http://aolserver.cvs.sourceforge.net/viewvc/aolserver/nsreturnz/nsreturnz.c

/* copy the gzip header into the first 10 bytes, wiping out the
first 2 bytes from compress2. Add 8 to outlen for the gzip
header (10-2), subtract 4 because we're ignoring the last 4
bytes of compress2 */


is there any reason you're not just using ns_returnz ?


cheers

Russell


> ns_register_proc GET /headers ad_headers_proc
>
> proc ad_headers_proc {ignore} {
> set result {}
> set headers [ns_conn headers]
> for {set i 0} {$i < [ns_set size $headers]} {incr i} {
> set key [ns_set key $headers $i]
> set value [ns_set value $headers $i]
> append result "$key: $value\n"
> }
> ns_updateheader Content-Encoding gzip
> ns_return 200 text/plain [ns_zlib gzip $result]
> }


--

Alexey Pechnikov

unread,
Dec 1, 2010, 5:34:58 AM12/1/10
to AOLS...@listserv.aol.com
> is there any reason you're not just using ns_returnz ?

There is no ns_returnz function in upstream AOL repository:

$ cd aolserver/
$ rgrep ns_returnz .


Best regards, Alexey Pechnikov.
http://pechnikov.tel/

russell muetzelfeldt

unread,
Dec 1, 2010, 5:42:13 AM12/1/10
to AOLS...@listserv.aol.com
On 01/12/2010, at 9:34 PM, Alexey Pechnikov wrote:

> > is there any reason you're not just using ns_returnz ?
>
> There is no ns_returnz function in upstream AOL repository:
>
> $ git clone git://github.com/aolserver/aolserver
> $ cd aolserver/
> $ rgrep ns_returnz .

hmmm, I guess it's one of the modules that Dossy hasn't copied over from sourceforge... other than that I'm not aware of any reason to not use it...

cheers

Russell

Alexey Pechnikov

unread,
Dec 1, 2010, 6:13:54 AM12/1/10
to AOLS...@listserv.aol.com
Well, I did see the code and write new Makefile:

MOD  =  nsreturnz
OBJS     =  nsreturnz.o
MODINIT  =  Ns_ModuleInit
include  ../include/ns.mak

The code is good... Imho this may be copied into new repository.

2010/12/1 russell muetzelfeldt <russm-a...@slofith.org>



--
Best regards, Alexey Pechnikov.
http://pechnikov.tel/

Alexey Pechnikov

unread,
Dec 1, 2010, 8:06:17 AM12/1/10
to AOLS...@listserv.aol.com
Can you help me to resolv the problem?

ns_returnz: compress2 failed, status=-5

--
Best regards, Alexey Pechnikov.
http://pechnikov.tel/

Tom Jackson

unread,
Dec 1, 2010, 10:45:03 AM12/1/10
to AOLS...@listserv.aol.com
Here's a temp link to a git version of the module:

http://junom.com/gitweb/gitweb.perl?p=aolserver.git;a=tree;f=nsreturnz;h=49d6471737b294;hb=master
(use the snapshot link on this page to get a zip or tgz of just this
module (unfortunately the enclosing directory will be misnamed).

I've already done a cvsimport on all the modules. I'll put them up on
github later today.

tom jackson

Tom Jackson

unread,
Dec 1, 2010, 10:51:11 AM12/1/10
to AOLS...@listserv.aol.com
BTW, I think the functionality in the external module nsreturnz is now
part of nszlib, which should be in your git repo, otherwise here's a
link:

http://junom.com/gitweb/gitweb.perl?p=aolserver.git;a=tree;f=nszlib;h=0ac26233;hb=origin

tom jackson

Alexey Pechnikov

unread,
Dec 1, 2010, 12:11:49 PM12/1/10
to AOLS...@listserv.aol.com
2010/12/1 Tom Jackson <t...@rmadilo.com>

BTW, I think the functionality in the external module nsreturnz is now
part of nszlib

But are you sure that this works? In my tests only ns_returnfile work correct for gzipped content:
 

ns_register_proc GET /test  ad_test_proc
proc ad_test_proc {ignore} {
    ns_updateheader Content-Encoding gzip
    ns_return 200 text/plain [ns_zlib gzip "test"]
}

$ curl http://localhost:8200/test 2>/dev/null| hexdump
0000000 c21f 088b 0000 0000 0000 2b03 2d49 012e
0000010 0c00 7f7e 98c3 0004 0000               
000001a


ns_register_proc GET /test  ad_test_proc
proc ad_test_proc {ignore} {
    set fd [open /tmp/test.gz r]
    fconfigure $fd -translation binary -encoding binary
    ns_updateheader Content-Encoding gzip
    ns_return 200 text/plain [read $fd]
    close $fd
}
$ echo -n test|gzip>/tmp/test.gz
$ curl http://localhost:8200/test 2>/dev/null| hexdump
0000000 c21f 088b c300 7f99 b6c3 004c 2b03 2d49
0000010 012e 0c00 7f7e 98c3 0004 0000          
000001c


ns_register_proc GET /test  ad_test_proc
proc ad_test_proc {ignore} {
    ns_updateheader Content-Encoding gzip
    ns_returnfile 200 text/plain /tmp/test.gz
}
$ echo -n test|gzip>/tmp/test.gz
$ curl http://localhost:8200/test 2>/dev/null| hexdump
0000000 8b1f 0008 7fd9 4cf6 0300 492b 2e2d 0001
0000010 7e0c d87f 0004 0000                    
0000018

Browsers can show only last question result. I don't understand how ns_zlib encode the data.

Tom Jackson

unread,
Dec 1, 2010, 12:42:02 PM12/1/10
to AOLS...@listserv.aol.com
I don't use this feature, it just puts extra work on the server and
client to save something. If service providers think it is necessary
to save bandwidth, they could probably compress the content
themselves. But from looking at the code, you shouldn't have to do
anything except configure the module to get transparent compression.
You have to set minimum file content size, etc. and adp needs
independent configuration if you use adps. In other words, you don't
have to edit your site code to get compression with ns_zlib.

tom jackson

Alexey Pechnikov

unread,
Dec 1, 2010, 1:23:11 PM12/1/10
to AOLS...@listserv.aol.com
Is possible transparent compression for ns_register_proc functions? I don't use adps in all my projects.

2010/12/1 Tom Jackson <t...@rmadilo.com>

I don't use this feature, it just puts extra work on the server and
client to save something. If service providers think it is necessary
to save bandwidth, they could probably compress the content
themselves. But from looking at the code, you shouldn't have to do
anything except configure the module to get transparent compression.
You have to set minimum file content size, etc. and adp needs
independent configuration if you use adps. In other words, you don't
have to edit your site code to get compression with ns_zlib.

tom jackson

Alexey Pechnikov

unread,
Dec 1, 2010, 1:19:21 PM12/1/10
to AOLS...@listserv.aol.com
This work correct:

ns_register_proc GET /test  ad_test_proc

proc ad_test_proc {ignore} {
    set gzip [ns_zlib gzip "test"]
    set time [ns_httptime [ns_time]]
    ns_write "HTTP/1.0 200 OK
Content-Type: text/plain; charset=utf-8
Content-Encoding: gzip
\n"
    ns_write $gzip
}
$ curl http://localhost:8200/test 2>/dev/null| hexdump
0000000 8b1f 0008 0000 0000 0300 492b 2e2d 0001
0000010 7e0c d87f 0004 0000                    
0000018


But is possible to disable all output translations by ns_return? Where is documented how ns_return process the data?..

Alexey Pechnikov

unread,
Dec 1, 2010, 2:48:19 PM12/1/10
to AOLS...@listserv.aol.com
But does not work for cyrillic strings. And zlib in Tcl 8.6 produce wrong result too:

    set content привет
    set gzip [binary format "H*iH*" "1f8b0800" [clock seconds] "0003"]
    append gzip [zlib deflate $content 0]
    append gzip [binary format i [zlib crc32 $content]]
    append gzip [binary format i [string length $content]]

    set fd [open /tmp/test.gz w]
    fconfigure $fd -translation binary -encoding binary
    puts -nonewline $fd $gzip
    close $fd

    puts [exec zcat /tmp/test.gz ]
    ?@825B


But for latin symbols all right:

    set content hello
    set gzip [binary format "H*iH*" "1f8b0800" [clock seconds] "0003"]
    append gzip [zlib deflate $content]
    append gzip [binary format i [zlib crc32 $content]]
    append gzip [binary format i [string length $content]]

    set fd [open /tmp/test.gz w]
    fconfigure $fd -translation binary -encoding binary
    puts -nonewline $fd $gzip
    close $fd

    puts [exec zcat /tmp/test.gz ]
    hello


2010/12/1 Alexey Pechnikov <pech...@mobigroup.ru>

Alexey Pechnikov

unread,
Dec 1, 2010, 3:30:31 PM12/1/10
to AOLS...@listserv.aol.com
There is the answer in the documentation for Tcl 8.6 

And so the correct solution is:

set gzip [ns_zlib gzip [encoding convertto utf-8 $content]]
ns_write "HTTP/1.0 200 OK
Content-Type: text/plain; charset=utf-8
Content-Encoding: gzip
\n"
ns_write $gzip

Tcl 8.6 has native zlib support:

set content [encoding convertto utf-8 $content]
set gzip [binary format "H*iH*" "1f8b0800" [clock seconds] "0003"]
append gzip [zlib deflate $content 0]
append gzip [binary format i [zlib crc32 $content]]
append gzip [binary format i [string length $content]]
ns_write "HTTP/1.0 200 OK
Content-Type: text/plain; charset=utf-8
Content-Encoding: gzip
\n"
ns_write $gzip


Can somebody to add these examples to AOL documentation?

Tom Jackson

unread,
Dec 1, 2010, 3:31:46 PM12/1/10
to AOLS...@listserv.aol.com
I must be missing something. What does the below code have to do with
AOLserver or ns_zlib?

Personally I also wouldn't assume that AOLserver works perfectly with
Tcl8.6. Unless you absolutely need 8.5, I would stick with the latest
8.4 version. For one thing 8.4 is faster than either.

tom jackson

2010/12/1 Alexey Pechnikov <pech...@mobigroup.ru>:


> But does not work for cyrillic strings. And zlib in Tcl 8.6 produce wrong
> result too:

> О©╫О©╫ О©╫set content О©╫О©╫О©╫О©╫О©╫О©╫
> О©╫О©╫ О©╫set gzip [binary format "H*iH*" "1f8b0800" [clock seconds] "0003"]
> О©╫О©╫ О©╫append gzip [zlib deflate $content 0]
> О©╫О©╫ О©╫append gzip [binary format i [zlib crc32 $content]]
> О©╫О©╫ О©╫append gzip [binary format i [string length $content]]
> О©╫О©╫ О©╫set fd [open /tmp/test.gz w]
> О©╫О©╫ О©╫fconfigure $fd -translation binary -encoding binary
> О©╫О©╫ О©╫puts -nonewline $fd $gzip
> О©╫О©╫ О©╫close $fd
> О©╫О©╫ О©╫puts [exec zcat /tmp/test.gz ]
> О©╫О©╫ О©╫?@825B


>
> But for latin symbols all right:

> О©╫О©╫ О©╫set content hello
> О©╫О©╫ О©╫set gzip [binary format "H*iH*" "1f8b0800" [clock seconds] "0003"]
> О©╫О©╫ О©╫append gzip [zlib deflate $content]
> О©╫О©╫ О©╫append gzip [binary format i [zlib crc32 $content]]
> О©╫О©╫ О©╫append gzip [binary format i [string length $content]]
> О©╫О©╫ О©╫set fd [open /tmp/test.gz w]
> О©╫О©╫ О©╫fconfigure $fd -translation binary -encoding binary
> О©╫О©╫ О©╫puts -nonewline $fd $gzip
> О©╫О©╫ О©╫close $fd
> О©╫О©╫ О©╫puts [exec zcat /tmp/test.gz ]
> О©╫О©╫ О©╫hello


>
> 2010/12/1 Alexey Pechnikov <pech...@mobigroup.ru>
>>
>> This work correct:

>> ns_register_proc GET /test О©╫ad_test_proc
>> proc ad_test_proc {ignore} {
>> О©╫О©╫ О©╫set gzip [ns_zlib gzip "test"]
>> О©╫О©╫ О©╫set time [ns_httptime [ns_time]]
>> О©╫О©╫ О©╫ns_write "HTTP/1.0 200 OK


>> Content-Type: text/plain; charset=utf-8
>> Content-Encoding: gzip
>> \n"

>> О©╫О©╫ О©╫ns_write $gzip

Jeff Hobbs

unread,
Dec 1, 2010, 3:43:53 PM12/1/10
to AOLS...@listserv.aol.com
On 01/12/2010 12:31 PM, Tom Jackson wrote:
> Personally I also wouldn't assume that AOLserver works perfectly with
> Tcl8.6. Unless you absolutely need 8.5, I would stick with the latest
> 8.4 version. For one thing 8.4 is faster than either.

That's no longer true.

Tom Jackson

unread,
Dec 1, 2010, 3:46:14 PM12/1/10
to AOLS...@listserv.aol.com
All you are doing here is using ns_zlib in Tcl. Who cares if this
works or not? It is possible that HTTP content compression is slightly
different than what is produced by ns_zlib. But this doesn't matter
really. What you should be testing is ns_returnz (sp?) or ns_return
(or adp output) with transparent compression (which takes into account
client capabilities, you can't just blindly send compressed content).

I remember looking into ns_zlib just because it was part of the adp
processing/fastpath code. I don't have any reason to believe that, if
configured correctly it doesn't work. I also doubt that ns_zlib would
have wound up in the core unless it worked, the author is well known
and I've never heard of anyone suggesting any of the repo modules
"don't work".

I'll try to look into how ns_zlib is supposed to work. Maybe this is a
total lack of documentation problem.

tom jackson

Hossein Sharifi

unread,
Dec 1, 2010, 5:20:26 PM12/1/10
to AOLS...@listserv.aol.com
I'm pretty sure that gzip compression is already natively and transparently supported in AOLserver as long as you build and include the ns_zlib module.  You shouldn't ever have to bother with encoding or calling a special function; rather, you just need to enable it in the config file: "ns_param gzip on" in the ns/server/${server} section of your config file (there's also a gzipmin param to set the minimum size required in bytes to trigger compression).

One problem is that AOLserver doesn't gzip content when ns_return is called; it works for pretty much every scenario except that one.  I don't know why this is the case; we just hacked this by recompiling nsd to always gzip regardless of the config file, as long as the client supports it (line 161 of connio.c).  It would be nice if this worked correctly by default (or if we figured out the proper way to enable its use in ns_return).
--
Hossein Sharifi
http://rateyourmusic.com

Tom Jackson

unread,
Dec 1, 2010, 8:13:50 PM12/1/10
to AOLS...@listserv.aol.com
2010/12/1 Jeff Hobbs <je...@activestate.com>:

> On 01/12/2010 12:31 PM, Tom Jackson wrote:
>>
>> Personally I also wouldn't assume that AOLserver works perfectly with
>> Tcl8.6. Unless you absolutely need 8.5, I would stick with the latest
>> 8.4 version. For one thing 8.4 is faster than either.
>
> That's no longer true.

Not sure what is no longer true. AOLserver is pretty much ahead of Tcl
wrt client/server programming (and about 15 year ahead of Java). Tcl
8.6 continues to head into the direction of poor performance and
useless new API. In case programmers haven't figured it out,
multi-core processors are the norm, so stuff like thread bound
coroutines are useless wrt performance and much harder to program than
C thread based servers. But if you are stuck in one thread,
maybe..."maybe" there is some benefit in moving to Tcl 8.6, unless you
are using AOLserver.

Please don't listen to the Tcl gurus, they are behind the
curve...actually they are actively slowing down the curve. The
evidence is simple: 8.4 is faster than 8.5 which is faster than 8.6.
How is this even possible? The answer is simple: the developers do no
know what they are doing. The evidence is the performance.

tom jackson

Gustaf Neumann

unread,
Dec 2, 2010, 3:44:04 AM12/2/10
to AOLS...@listserv.aol.com
On 01.12.10 21:31, Tom Jackson wrote:
> Personally I also wouldn't assume that AOLserver works perfectly with
> Tcl8.6.
Me neither, but Tcl 8.6 is still in an early state.

> Unless you absolutely need 8.5, I would stick with the latest
> 8.4 version. For one thing 8.4 is faster than either.
Well, it depends. My recommendation is to use Tcl 8.5.
From our experience from running large OpenACS apps
(600.000 loc Tcl on our production environment, delivering
up to 220 GB/day, up to 2.500+ concurrent users), the
performance difference between 8.4 and 8.5 is very little
as long one is using just 8.4 functionality. Tcl 8.5 has several
improvements to make programs more readably (e.g.
the "in" operator in expressions), more safe (e.g. the expand
operator) and leads to more stable multi-threaded programs
(e.g. c-stack protection, checking stack boundaries instead
of plain crashing as in 8.4), provides bignums, and many
many more really useful fixes and improvements.

The OpenACS community has decided some time ago
to require 8.5 for future releases.

>Please don't listen to the Tcl gurus, they are behind the
>curve...actually they are actively slowing down the curve.

Come on, stop that bashing. Jeff was referring to
http://code.activestate.com/lists/tcl-core/9805/

On real problems, the Tcl-core people are very helpful
and highly responsive...

-gustaf neumann

Jeff Hobbs

unread,
Dec 2, 2010, 10:42:41 AM12/2/10
to AOLS...@listserv.aol.com
On 2010-12-02, at 12:44 AM, Gustaf Neumann wrote:

> >Please don't listen to the Tcl gurus, they are behind the
> >curve...actually they are actively slowing down the curve.
>
> Come on, stop that bashing. Jeff was referring to
> http://code.activestate.com/lists/tcl-core/9805/

Actually even more recent changes have improved things further. 8.5 is now actually faster than 8.4 on the benchmarks. In any case, we've learned to tune out people that only throw heat without light.

Jeff

Alexey Pechnikov

unread,
Dec 6, 2010, 12:13:22 PM12/6/10
to AOLS...@listserv.aol.com
I'm see the code in connio.c:

    /*
     * GZIP the content when not streaming if enabled and the content
     * length is above the minimum.
     */

    if (!stream
   && (conn->flags & NS_CONN_GZIP)
   && (servPtr->opts.flags & SERV_GZIP)
   && (len > (int) servPtr->opts.gzipmin)
   && (ahdr = Ns_SetIGet(conn->headers, "Accept-Encoding")) != NULL
   && strstr(ahdr, "gzip") != NULL
   && Ns_Gzip(buf, len, servPtr->opts.gziplevel, &gzip) == NS_OK) {
buf = gzip.string;
len = gzip.length;
Ns_ConnCondSetHeaders(conn, "Content-Encoding", "gzip");
    }

There are no checks for content-type and older version of Internet Explorer (IE5, IE6 and may be IE7 have a lot of problems with gzipped scripts and styles). I don't think that this code is useful for production. And we may add ETAG functionality and smart caching checksums of results for decreasing data transfer and server loading. I dont know about your situation but my clients have limited internet connections (especially on mobile devices) and ETAG header transmitting is faster when gzipped content... For static files on group of hosts application-defined ETAG is helpful too but internal AOL last-modified-since mechanizm is niot useful (it's not the AOL problem, of cource).

P.S. I dont understand why my suggestion to complete AOL documentation by examples was produce the holywar about Tcl 8.4 vs 8.5 vs 8.6. I think AOL 4.5.1 + Tcl 8.5 is better choice for new projects and Tcl 8.6 is better for some utilities (fast internal base64 realization, half-closed sockets and other features help me to build faster applications with a few lines of code). Tcl 8.6 documentation of zlib functions is much better than AOL documentation of ns_zlib module and some of this docs and examples can be helpful for AOL, why not?


2010/12/2 Hossein Sharifi <sha...@rateyourmusic.com>

Tom Jackson

unread,
Dec 6, 2010, 5:51:45 PM12/6/10
to AOLS...@listserv.aol.com
On Mon, Dec 6, 2010 at 9:13 AM, Alexey Pechnikov <pech...@mobigroup.ru> wrote:
> I'm see the code in�connio.c:
> �� �/*
> �� � * GZIP the content when not streaming if enabled and the content
> �� � * length is above the minimum.
> �� � */
> �� �if (!stream
> � �&& (conn->flags & NS_CONN_GZIP)
> � �&& (servPtr->opts.flags & SERV_GZIP)
> � �&& (len > (int) servPtr->opts.gzipmin)
> � �&& (ahdr = Ns_SetIGet(conn->headers, "Accept-Encoding")) != NULL
> � �&& strstr(ahdr, "gzip") != NULL
> � �&& Ns_Gzip(buf, len, servPtr->opts.gziplevel, &gzip) == NS_OK) {
> buf = gzip.string;
> len = gzip.length;
> Ns_ConnCondSetHeaders(conn, "Content-Encoding", "gzip");
> �� �}
> There are no checks for content-type and older version of Internet Explorer
> (IE5, IE6 and may be IE7 have a lot of problems with gzipped scripts and
> styles).

And yet your examples provided even less customization. There is
almost no reason to waste cpu on compressing output, just provide a
gzipped file for very large files. Who are you trying to save money
for anyway?

> I don't think that this code is useful for production.

Right, then don't use it.

> And we may
> add ETAG functionality and smart caching checksums of results for decreasing
> data transfer and server loading. I dont know about your situation but my
> clients have limited internet connections (especially on mobile devices) and
> ETAG header transmitting is faster when gzipped content...

Then the least of your problems is gzipping content, you need to
actively minimize the data transfered. But all of this sounds like
_your_ problem, not the problem of a generic application server. You
haven't even figured out how automatic compression works in AOLserver
yet you want to propose additional features.

> For static files
> on group of hosts application-defined ETAG is helpful too but internal AOL
> last-modified-since mechanizm is niot useful (it's not the AOL problem, of
> cource).

??

> P.S. I dont understand why my suggestion to complete AOL documentation by
> examples was produce the holywar about Tcl 8.4 vs 8.5 vs 8.6.

Because I'm a jerk and overreact to what I think are idiotic statements?

BTW, you didn't provide any useful additions to AOLserver documentation.

> I think AOL
> 4.5.1 + Tcl 8.5 is better choice for new projects and Tcl 8.6 is better for
> some utilities (fast internal base64 realization, half-closed sockets and
> other features help me to build faster applications with a few lines of
> code).

I agree with Gustaf: the latest 8.5 is worth the effort. There are
certain features which simplify very annoying code. This is true even
if your version of 8.5 is slower than 8.4. But you have to actively
update your code to take advantage of the new features. The more code
you have, the less benefit you get from upgrading without code
conversion. However, Gustaf mentioned a higher stability in 8.5. This
could easily override the limited benefit of simply moving the Tcl
library to 8.5 from 8.4.

> Tcl 8.6 documentation of zlib functions is much better than AOL
> documentation of ns_zlib module and some of this docs and examples can be
> helpful for AOL, why not?

If you use Tcl, use the Tcl documentation. If you use AOLserver, use
the AOLserver documentation. I'm not sure why you keep confusing these
two things.

tom jackson (AKA the jerk)

Jeff Rogers

unread,
Dec 6, 2010, 6:40:47 PM12/6/10
to AOLS...@listserv.aol.com
Hossein Sharifi wrote:
> One problem is that AOLserver doesn't gzip content when ns_return is
> called; it works for pretty much every scenario except that one. I
> don't know why this is the case; we just hacked this by recompiling nsd
> to always gzip regardless of the config file, as long as the client
> supports it (line 161 of connio.c). It would be nice if this worked
> correctly by default (or if we figured out the proper way to enable its
> use in ns_return).

I only skimmed over the code, but it looks like the problem is that the
content is only gzipped if the NS_CONN_GZIP flag is set on the conn, and
the only place that flag is set is in the adp processor.

The obvious (I think) improvement is to expose
Ns_ConnGetGzipFlag/Ns_ConnSetGzipFlag as subcommands of ns_conn, which
would let you write a filter to do the ancient/broken browser checking
or whatever else you needed and turn gzip on or off as needed.
(However, doing this could impact adp processing, so a corresponding
change might need to be made there, lest your adp code has a different
idea about what should or shouldn't be compressed)

-J

Brett Schwarz

unread,
Dec 7, 2010, 12:17:09 AM12/7/10
to AOLS...@listserv.aol.com

We had a similar discussion about this a couple years ago. See
http://www.mail-archive.com/aols...@listserv.aol.com/msg11598.html (although
the threading is messed up in the mail archive, so you will have to skip
around).


At the time, if I remember, there wasn't a real consensus, so I made changes in
my sandbox to mimic the adp compression but for ns_return. If people are
interested, I can try to dig up the code.

Mark Aufflick

unread,
Dec 19, 2010, 5:39:58 AM12/19/10
to AOLS...@listserv.aol.com
Hi Tom,

Notwithstanding your legitimate issue that gzipping every html and css
file on the fly is counterproductive in many cases, one case this is
not true is serving to mobile devices - if you're on the end of a weak
GPRS connection with a fairly powerful cpu, you are going to notice
the difference between gzipped and ungzipped content.

Just my 2c :)
--
Mark Aufflick
http://mark.aufflick.com/about/contact
http://pumptheory.com/about

Alexey Pechnikov

unread,
Dec 19, 2010, 6:54:23 AM12/19/10
to AOLS...@listserv.aol.com
Hello!

I did test the gzipping on-the-fly on 2x Intel Xeon 5550 and the output stream may be up to 3 Gigabit/s on localhost. With 100 Megabit internet connection the zlib-compression is very nice for production :-) We may be limited by internet bandwidth but not by CPU.

2010/12/19 Mark Aufflick <mark-ao...@aufflick.com>

--
Best regards, Alexey Pechnikov.
http://pechnikov.tel/

Tom Jackson

unread,
Dec 19, 2010, 12:39:01 PM12/19/10
to AOLS...@listserv.aol.com
True, but since gzip is something which can be applied at any point
between the source and destination, you have to wonder why the mobile
service provider isn't doing this for their customers.

If your website/service actually serves up content for mobile
customers, it would also help to generate scaled down content for
these users. This is a much bigger change than simply compressing the
content, but many web pages seem unusable in on small device screens,
heck some of the triple column news sites are almost impossible on a
laptop.

Maybe we should look at putting this into ns_return/fastpath if Brett
has a list of the changes.

tom jackson

Brett: you could email me the changed source files instead of
providing a patch to some git or cvs version.

Reply all
Reply to author
Forward
0 new messages