Re: [AOLSERVER] Does not work ns_return + zlib

[Peter Sadlon]

Did the original author find a solution to this?

I remember reading about a similar issue a while back, and always kept it in my mind for a reason.  Anyways, when TCL is compiled, by default it uses charset iso8859-1 and this can cause some issues, so what I had read was that you should recompile TCL to use UTF-8 by default.  That being said, I have no idea where I read that.

But as I am now working on a multi-lingual site, I have begun to revisit this issue and I think I found something that may be of interest since I believe it may also be what was causing the original poster some issues.

So lets assume I have a filter like this:

ns_register_filter preauth GET /test.html decode_url
proc decode_url { why } {
  ns_updateheader "Content-Encoding" "gzip"
  set page [GENERATE YOUR PAGE HERE]
  set page [encoding convertto "utf-8" $page]
  set page [ns_zlib gzip $page]

  ns_log notice [string length $page]

  set fd [open /tmp/test.gz w]
  fconfigure $fd -translation binary -encoding binary
  puts -nonewline $fd $page
  close $fd

  ns_returnfile 200 "$mime" "/tmp/test.gz"
#  ns_return 200 "$mime" "$page"

  return filter_return
}

This will work....
Notice I log the string length of $page, because, if I go to http://web-sniffer.net/ to get a view of the headers in the request and response I can see that the Content-Length in the response is the same as what gets logged.

However, if I now comment out the ns_returnfile and uncomment the ns_return line and visit web-sniffer again, I see I have a much larger Content-Length, which is why the encoding appears to fail.

ns_return is NsTclReturnObjCmd which in turn calls Tcl_GetStringFromObj(objv[objc-1], &len); which contains:
    if (lengthPtr != NULL) {
        *lengthPtr = objPtr->length;
    }
which I believe objPtr->length is returning the iso8859-1 length and not the utf-8 length.

But I will have to leave that to someone smarter to figure out.

Another solution is of course just to ns_write out the headers:

ns_write "HTTP/1.1 200 OK
Expires: now
MIME-Version: 1.0
Date: [ns_httptime [ns_time]]
Content-Type: text/plain; charset=utf-8
Content-Encoding: gzip
Content-Length: [string length $page]
\n$page"

instead of using the ns_return function (thus being able to gzip without having to store the zipped file)



> Date: Sun, 19 Dec 2010 09:39:01 -0800
> From: t...@RMADILO.COM
> Subject: Re: [AOLSERVER] Does not work ns_return + zlib
> To: AOLS...@LISTSERV.AOL.COM
>
> True, but since gzip is something which can be applied at any point
> between the source and destination, you have to wonder why the mobile
> service provider isn't doing this for their customers.
>
> If your website/service actually serves up content for mobile
> customers, it would also help to generate scaled down content for
> these users. This is a much bigger change than simply compressing the
> content, but many web pages seem unusable in on small device screens,
> heck some of the triple column news sites are almost impossible on a
> laptop.
>
> Maybe we should look at putting this into ns_return/fastpath if Brett
> has a list of the changes.
>
> tom jackson
>
> Brett: you could email me the changed source files instead of
> providing a patch to some git or cvs version.
>
> On Sun, Dec 19, 2010 at 2:39 AM, Mark Aufflick
> <mark-ao...@aufflick.com> wrote:
> > Hi Tom,
> >
> > Notwithstanding your legitimate issue that gzipping every html and css
> > file on the fly is counterproductive in many cases, one case this is
> > not true is serving to mobile devices - if you're on the end of a weak
> > GPRS connection with a fairly powerful cpu, you are going to notice
> > the difference between gzipped and ungzipped content.
> >
> > Just my 2c :)
> > --
> > Mark Aufflick
> >  http://mark.aufflick.com/about/contact
> >  http://pumptheory.com/about
> >
> >
> >
> >
> > On Tue, Dec 7, 2010 at 9:51 AM, Tom Jackson <t...@rmadilo.com> wrote:
> >> On Mon, Dec 6, 2010 at 9:13 AM, Alexey Pechnikov <pech...@mobigroup.ru> wrote:
> >>> I'm see the code in connio.c:
> >>>     /*
> >>>      * GZIP the content when not streaming if enabled and the content
> >>>      * length is above the minimum.
> >>>      */
> >>>     if (!stream
> >>>    && (conn->flags & NS_CONN_GZIP)
> >>>    && (servPtr->opts.flags & SERV_GZIP)
> >>>    && (len > (int) servPtr->opts.gzipmin)
> >>>    && (ahdr = Ns_SetIGet(conn->headers, "Accept-Encoding")) != NULL
> >>>    && strstr(ahdr, "gzip") != NULL
> >>>    && Ns_Gzip(buf, len, servPtr->opts.gziplevel, &gzip) == NS_OK) {
> >>> buf = gzip.string;
> >>> len = gzip.length;
> >>> Ns_ConnCondSetHeaders(conn, "Content-Encoding", "gzip");
> >>>     }
> >>> There are no checks for content-type and older version of Internet Explorer
> >>> (IE5, IE6 and may be IE7 have a lot of problems with gzipped scripts and
> >>> styles).
> >>
> >> And yet your examples provided even less customization. There is
> >> almost no reason to waste cpu on compressing output, just provide a
> >> gzipped file for very large files. Who are you trying to save money
> >> for anyway?
> >>
> >>> I don't think that this code is useful for production.
> >>
> >> Right, then don't use it.
> >>
> >>> And we may
> >>> add ETAG functionality and smart caching checksums of results for decreasing
> >>> data transfer and server loading. I dont know about your situation but my
> >>> clients have limited internet connections (especially on mobile devices) and
> >>> ETAG header transmitting is faster when gzipped content...
> >>
> >> Then the least of your problems is gzipping content, you need to
> >> actively minimize the data transfered. But all of this sounds like
> >> _your_ problem, not the problem of a generic application server. You
> >> haven't even figured out how automatic compression works in AOLserver
> >> yet you want to propose additional features.
> >>
> >>> For static files
> >>> on group of hosts application-defined ETAG is helpful too but internal AOL
> >>> last-modified-since mechanizm is niot useful (it's not the AOL problem, of
> >>> cource).
> >>
> >> ??
> >>
> >>> P.S. I dont understand why my suggestion to complete AOL documentation by
> >>> examples was produce the holywar about Tcl 8.4 vs 8.5 vs 8.6.
> >>
> >> Because I'm a jerk and overreact to what I think are idiotic statements?
> >>
> >> BTW, you didn't provide any useful additions to AOLserver documentation.
> >>
> >>> I think AOL
> >>> 4.5.1 + Tcl 8.5 is better choice for new projects and Tcl 8.6 is better for
> >>> some utilities (fast internal base64 realization, half-closed sockets and
> >>> other features help me to build faster applications with a few lines of
> >>> code).
> >>
> >> I agree with Gustaf: the latest 8.5 is worth the effort. There are
> >> certain features which simplify very annoying code. This is true even
> >> if your version of 8.5 is slower than 8.4. But you have to actively
> >> update your code to take advantage of the new features. The more code
> >> you have, the less benefit you get from upgrading without code
> >> conversion. However, Gustaf mentioned a higher stability in 8.5. This
> >> could easily override the limited benefit of simply moving the Tcl
> >> library to 8.5 from 8.4.
> >>
> >>> Tcl 8.6 documentation of zlib functions is much better than AOL
> >>> documentation of ns_zlib module and some of this docs and examples can be
> >>> helpful for AOL, why not?
> >>
> >> If you use Tcl, use the Tcl documentation. If you use AOLserver, use
> >> the AOLserver documentation. I'm not sure why you keep confusing these
> >> two things.
> >>
> >> tom jackson (AKA the jerk)
> >>
> >>
> >> --
> >> AOLserver - http://www.aolserver.com/
> >>
> >> To Remove yourself from this list, simply send an email to <list...@listserv.aol.com> with the
> >> body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
> >>
> >
> >
> > --
> > AOLserver - http://www.aolserver.com/
> >
> > To Remove yourself from this list, simply send an email to <list...@listserv.aol.com> with the
> > body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
> >
>
>
> --
> AOLserver - http://www.aolserver.com/
>
> To Remove yourself from this list, simply send an email to <list...@listserv.aol.com> with the
> body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.

--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to <list...@listserv.aol.com> with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.

[Alexey Pechnikov]

Hello!

I use this code now:

ns_register_proc GET *.css  ad_gzip

ns_register_proc GET *.js   ad_gzip

ns_register_proc GET *.html ad_gzip

proc ad_gzip {} {

    set url [ns_conn url]

    set fname [ns_url2file $url]

    if { [file exists $fname] == 0 } {

        ns_returnnotfound

        return

    }

    set mtime [file mtime $fname]

    set size [file size $fname]

    set last  [ns_set iget [ns_conn headers] "If-Modified-Since"]

    if { $last ne {} && [clock scan $last] >= $mtime } {

        ns_write "HTTP/1.0 304 Not Modified\n\n"

    } elseif { $size > 1024 && [string first gzip [ns_set iget [ns_conn headers] "Accept-Encoding"]] >= 0 } {

        set fd [open $fname r]

        set content [read $fd]

        close $fd

        set gzip [ns_zlib gzip [encoding convertto utf-8 $content]]

        set time [ns_time]

        ns_write "HTTP/1.0 200 OK

Content-Type: [ns_guesstype $fname]

Content-Encoding: gzip

Last-Modified: [ns_httptime $mtime]

Date: [ns_httptime $time]

Content-Length: [string length $gzip]

\n"

        ns_write $gzip

    } else {

        ns_returnfile 200 [ns_guesstype $fname] $fname

    }

}

2011/1/14 Peter Sadlon <f_pet...@hotmail.com>

--
Best regards, Alexey Pechnikov.
http://pechnikov.tel/