[AOLSERVER] nsdci query

[Damien O'Rourke]

Hi,

I've been experimenting with the components in nsdci, most
particularly the sob service, and am impressed with it. What I'm
struggling with though is if I store an image file to the sob with
nsob.copy and then try and retrieve it later with nsob.get, it only
returns the first 4 or 10 bytes of the file. If I do a nsob.put I
can then do a nsob.get successfully, but the image file stored by the
nsob.put command is not viewable with other image viewers. I suspect
this is an encoding/binary issue but haven't been able to figure out
the where and whyfor.
Any ideas on where the issue might be or whether it's even possible
to do what I want using the sob component?

Thanks

Damien O'Rourke

--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to <list...@listserv.aol.com> with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.

[Dossy Shiobara]

On 2007.10.23, Damien O'Rourke <doro...@CARSALES.COM.AU> wrote:
> I've been experimenting with the components in nsdci, most
> particularly the sob service, and am impressed with it.

I'm SO glad you were able to get it working! I keep hearing "we need
documentation first" from people, but apparently there was enough for
you to get it going, at least.

> What I'm struggling with though is if I store an image file to the

> sob with sob.copy and then try and retrieve it later with nsob.get,

> it only returns the first 4 or 10 bytes of the file.

SOB isn't encoding-aware and is actually really naive--it doesn't handle
binary data well.

A work-around that may work is to use Tcl's [encoding] explicitly,
i.e.:

nsob.put $id [encoding convertto utf-8 $value]

I believe if you do this, you shouldn't need to do anything special when
you nsob.get, as Tcl internally expects the data to be utf-8.

-- Dossy

--
Dossy Shiobara | do...@panoptic.com | http://dossy.org/
Panoptic Computer Network | http://panoptic.com/
"He realized the fastest way to change is to laugh at your own
folly -- then you can let go and quickly move on." (p. 70)

[Michael Andrews]

nsob is mostly used for text content. While the "copy" would work
with a binary file - a "get" would probably not work with a binary
file since it is set up over HTTP and the binary code might terminate
the HTTP response before intended. But that is just a guess. I have
not really read the RPC/HTTP code in nsdci.

M

[Damien O'Rourke]

Dossy,

thanks for the quick response. I didn't find any issues in getting it
working ... even splitting the components across different servers.
And possibly I didn't explain my issue adequately as what I can do
successfully at the moment is
- have an image uploaded via web page to frontend server (which saves
it as temp file)
- open and load the temp file as binary into a variable and send that
to the sob server with nsob.put
-- This stores it ... but not in binary format (i believe) as I can't
view that image with some other image viewer, but I can view it in
web page using nsob.get. So I think it's already in utf-8 format,
which is why it works.

But what doesn't work is
- open and load the temp file as binary into a variable and send that
to the sob server with nsob.copy
-- This stores it ... and in the original binary format as it's
viewable with other image viewers. But is not viewable using nsob.get

I really need to keep the binary format when it's stored by the sob
server ... so the images can be viewed by other apps ... and also be
viewable using nsob.get. Where it seems to be failing at the moment
is, even though the sob server stored the data as binary (using
nsob.copy), it doesn't seem to be able to read it in and return it
using nsob.get. So I think nsob.get needs to be able to convert from
binary to utf-8 to successfully return it ... ??

Thanks

Damien O'Rourke
doro...@carsales.com.au

[Jeff Rogers]

Dossy Shiobara wrote:
>> What I'm struggling with though is if I store an image file to the
>> sob with sob.copy and then try and retrieve it later with nsob.get,
>> it only returns the first 4 or 10 bytes of the file.
>
> SOB isn't encoding-aware and is actually really naive--it doesn't handle
> binary data well.

Not sure if this it related, but I was just looking at ns_http for an
unrelated purpose and it appears that it is not binary clean at all,
since it uses Tcl_SetVar which expects a null-terminated string.
Getting binary files typically resulted in a 4 byte response, could be
that an average jpg file has a null at byte 4.

-J

[Damien O'Rourke]

Thanks Andrew. And what Jeff Rogers mentioned about ns_http not being
binary friendly seems to tie in with that.

The way I've got round it so far is to use the nsob.copy for loading
up the files, and nproxy plus 2 level cache like the sob system for
getting the image files. That way I can at least set the encoding
when reading and returning. It would have been nicer if the sob
system handled it all though ...

Damien

On 23/10/2007, at 10:47 AM, Michael Andrews wrote:

> nsob is mostly used for text content. While the "copy" would work
> with a binary file - a "get" would probably not work with a binary
> file since it is set up over HTTP and the binary code might
> terminate the HTTP response before intended. But that is just a
> guess. I have not really read the RPC/HTTP code in nsdci.
>
> M
>
> On Oct 22, 2007, at 7:28 PM, Damien O'Rourke wrote:
>

[Michael Andrews]

That's what I believe as well. While ns_http is a different code base
- I think both suffer from the same binary termination issue. I'm not
sure if we saw this when using nsob with the old DCIRPC protocol -
but we never used it for binary, so when we moved to HTTP we probably
never thought it to be problematic.

Check out dcicommon/rpc.c and dcicommon/server.c (I think).

M

[Tom Jackson]

Jeff,

I have ns_http working, at least for me here with binary data. Supposedly this
is the reason for ns_http (not ns_httpget/post or ns_httpsget/post):

Here is a script:

set id [ns_http queue http://192.168.1.102:8000/sns-thumb.jpg]

set ok [ns_http wait -result image -status status $id]

if {"$status" == "200"} {
set fd [open [file join [file dirname [info script]] myimage.jpg] w+]
} else {
ns_return $status text/html $image
return -code return
}

# Very important:
fconfigure $fd -translation binary

puts $fd $image

close $fd

ns_return $status text/html "Status: $status <br>
<a href=\"/myimage.jpg\">My Image</a>"

[Michael Andrews]

I think the ns_http bug was fixed in the HEAD by Nate.

[Rick Cobb]

It seems to me like in AOLServer 4, we have four ways for Tcl code to
behave as an HTTP client. Has anybody done any compare & contrast on
them? As far as I can tell without a lot of code experience on some of
them, we have:

ns_http{open,get,post}
Generally synchronous
Only supports GET and POST
Only supports BASIC auth
No proxy support
No SSL (https) support
Implemented entirely in Tcl
Best compatibility (available in AOLServer 2 - 4)
Built-in
Only usable from a single thread (no channel/socket passing)

ns_http (http://panoptic.com/wiki/aolserver/Ns_http)
Supports asynchronous operation
Supports all methods
Only supports BASIC auth
No proxy support
No SSL (https) support
Implemented largely in C
Available only in AOLServer 4 and above
Built-in
Only usable from a single thread (no channel/socket passing)

Tcl's HTTP package (the one I know the least about)

Supports asynchronous operation with progress callbacks
Supports GET, POST, and HEAD
Not obvious how to use any authentication other than BASIC
Supports proxy on a per-call basis (but you're responsible for .pac
translation if necessary).
Supports TLS/SSL (via callback)
Somewhat more complicated API to use than the above options
Built-in since Tcl 7?

TclCurl
(http://personal1.iddeo.es/andresgarci/tclcurl/english/docs.html)
Supports asynchronous operation
Supports GET and POST, not clear about others but don't look
supported
Supports BASIC, DIGEST, GSS-Negotiate, NTLM
Supports proxy on a per-call basis (but you're responsible for .pac
translation if necessary)
Implemented in C
Requires packaging work (bringing it into your Tcl & AOLServer
build)
Multi-threading requires config / build - time work (c-ares)
Passing handle between threads requires wrapping in your own mutex
Supports FTP, telnet, other common URLs
Substantially more complicated API to use than the other options

So, for example, if I want to communicate with a RESTful application
(i.e., uses PUT, DELETE, etc) over HTTPS, it looks like there is no
option. Of course, most such applications also provide a POST wrapper,
but didn't the way-back Navi/AOLServer team have a solution for this?

Thanks --
-- ReC

[Jeff Rogers]

Michael Andrews wrote:
> I think the ns_http bug was fixed in the HEAD by Nate.

*ashamed* Yeah, the code I'm looking at is 4.0.8, pretty old. I
probably just need to upgrade and this will work better.

[Dossy Shiobara]

On 2007.10.23, Rick Cobb <rc...@KNOWNOW.COM> wrote:
> So, for example, if I want to communicate with a RESTful application
> (i.e., uses PUT, DELETE, etc) over HTTPS, it looks like there is no
> option. Of course, most such applications also provide a POST wrapper,
> but didn't the way-back Navi/AOLServer team have a solution for this?

As part of the nsopenssl module, there are (IMHO, broken)
implementations of ns_httpsopen and friends.

I think TclCurl will do SSL if the underlying libcurl is compiled with
support for it, too.

-- Dossy

--
Dossy Shiobara | do...@panoptic.com | http://dossy.org/
Panoptic Computer Network | http://panoptic.com/
"He realized the fastest way to change is to laugh at your own
folly -- then you can let go and quickly move on." (p. 70)

[Tom Jackson]

On Tuesday 23 October 2007 09:07, Rick Cobb wrote:
> ns_http (http://panoptic.com/wiki/aolserver/Ns_http)
>      Supports asynchronous operation
>      Supports all methods
>      Only supports BASIC auth
>      No proxy support
>      No SSL (https) support
>      Implemented largely in C
>      Available only in AOLServer 4 and above
>      Built-in
>      Only usable from a single thread (no channel/socket passing)

This isn't quite accurate.

What is going on in ns_http, is that a single thread does socket operations.
When you do a ns_http queue, it places the request in a queue in this thread
(different than a connection thread).

The [ns_http queue] returns an id used during [ns_http wait]. [ns_http wait]
can be called in a separate thread, like a scheduled proc. So you don't have
to pass a socket around, it always remains in the queue thread.

Also, you can add headers in a similar way to a regular request. That is, you
provide a ns_set with additional headers. So whatever auth methods are
supported with headers, you can do with ns_http.

There is also ns_httpsget/post which handles https, just like ns_httpget/post.
The commands ns_httpopen/ns_httpsopen are use by the get/post indirectly. I
also think these may follow redirects.

tom jackson

[Damien O'Rourke]

So ns_http handles binary but sob doesn't ... but sob could if it
used similar binary handling code to ns_http?

Damien

[Jay Rohr]

In general, I would not recommend using sob. It was developed for very
specific requirements at AOL and it does not scale well under load -
volume or frequency. We are in the process of replacing it with
something else.

Jay

[Dossy Shiobara]

On 2007.10.25, Jay Rohr <jay...@VERIZON.NET> wrote:
> In general, I would not recommend using sob. It was developed for very
> specific requirements at AOL and it does not scale well under load -
> volume or frequency. We are in the process of replacing it with
> something else.

Okay, this is just plain misinformation. SOB works very well for what
it was intended for, a low-write, high-read, distributed service for
small objects (thus, the name "SOB").

Any scaling issues AOL is having has to do with how it has since been
misused and/or operationalized.

-- Dossy

--
Dossy Shiobara | do...@panoptic.com | http://dossy.org/
Panoptic Computer Network | http://panoptic.com/
"He realized the fastest way to change is to laugh at your own
folly -- then you can let go and quickly move on." (p. 70)

[Tom Jackson]

On Thursday 25 October 2007 07:19, Dossy Shiobara wrote:
> On 2007.10.25, Jay Rohr <jay...@VERIZON.NET> wrote:
> > In general, I would not recommend using sob. It was developed for very
> > specific requirements at AOL and it does not scale well under load -
> > volume or frequency. We are in the process of replacing it with
> > something else.
>
> Okay, this is just plain misinformation. SOB works very well for what
> it was intended for, a low-write, high-read, distributed service for
> small objects (thus, the name "SOB").

I would agree with this. Although I haven't used SOB, I have looked at the
code. If anyone can improve upon reading static information from disk or
memory, then they can outperform SOB. My understanding is that SOB
simply 'compiles' dynamic information into a semi-static form. For instance,
how often does the weather change in Seattle? Although the information is
dynamic over large timescales, over the course of one day, not much changes.
If you have 100 queries a second for Seattle weather, maybe put it into a SOB
and let a backend service update it when needed.

Just because you have dynamic data doesn't mean you have to shoot yourself in
the foot when your service becomes popular.

I wrote a small subscribe/push service that for some reason I called a
tclbean. (I gave up waiting for NVs to be released by AOL). Essentially a
server (as client) connects to the publisher and gives a callback address and
a series of nsv arrays to subscribe to. The publish server connects back to
the client. A scheduled proc on the publish server pushes updates. The
original client can then always grab information from the nsv array and not
worry about anything else. It is hard to imagine scalability problems with
design, and I doubt there are any with SOB. But it would be interesting to
hear what issues were found.

tom jackson

[Jay Rohr]

Dossy Shiobara wrote:
> On 2007.10.25, Jay Rohr <jay...@VERIZON.NET> wrote:
>
>> In general, I would not recommend using sob. It was developed for very
>> specific requirements at AOL and it does not scale well under load -
>> volume or frequency. We are in the process of replacing it with
>> something else.
>>
>
> Okay, this is just plain misinformation. SOB works very well for what
> it was intended for, a low-write, high-read, distributed service for
> small objects (thus, the name "SOB").
>
> Any scaling issues AOL is having has to do with how it has since been
> misused and/or operationalized.
>
>

That was my point :-)

> -- Dossy

[Michael Andrews]

In general, I find the SOB model works well. The caching on the
server and the client provide unmatched performance. The NCF provides
a mechanism to flush the cache on updates. The Devil is always in the
details. How you set up multiple SOB servers and then get the
publishes to multiple SOB servers can certainly create scaling and
topology management issues.

Most sites to do not have the demands that AOL has - and a simple SOB
solution would be nice addition to any content distribution system.
However there are advantages to stateless systems. :-) (How's that
for diplomatic)

M

[John Caruso]

On Tuesday 10:30 AM 10/23/2007, Dossy Shiobara wrote:
>As part of the nsopenssl module, there are (IMHO, broken)
>implementations of ns_httpsopen and friends.

Could you expand on what you think is broken about them?

- John

[Dossy Shiobara]

On 2007.10.25, John Caruso <jca...@ARENASOLUTIONS.COM> wrote:
> On Tuesday 10:30 AM 10/23/2007, Dossy Shiobara wrote:
> >As part of the nsopenssl module, there are (IMHO, broken)
> >implementations of ns_httpsopen and friends.
>
> Could you expand on what you think is broken about them?

At AOL, a team that was using them had run into some kind of hang issue
where a blocking Tcl [read] was done on the Tcl channel (that was backed
by an nsopenssl SSL conn) that would never wake up and/or would time
out.

-- Dossy

--
Dossy Shiobara | do...@panoptic.com | http://dossy.org/
Panoptic Computer Network | http://panoptic.com/
"He realized the fastest way to change is to laugh at your own
folly -- then you can let go and quickly move on." (p. 70)

[Damien O'Rourke]

It was the two tier caching for performance and the auto flush that appealed to me ... serving lots of thumbnail images which occasionally get updated, and over time get outdated and unused, being replaced in use, and cache, by others.
Pity then sob doesn't do binary ...

Any advice from experience on nproxy in relation to scaling and load

Damien

-----Original Message-----
From: AOLserver Discussion on behalf of Michael Andrews
Sent: Fri 10/26/2007 2:12 AM
To: AOLS...@LISTSERV.AOL.COM
Subject: Re: [AOLSERVER] nsdci query

[Dossy Shiobara]

On 2007.10.26, Damien O'Rourke <doro...@CARSALES.COM.AU> wrote:
> It was the two tier caching for performance and the auto flush that
> appealed to me ... serving lots of thumbnail images which occasionally
> get updated, and over time get outdated and unused, being replaced in
> use, and cache, by others.
>
> Pity then sob doesn't do binary ...

Enhancing SOB to properly round-trip binary data isn't unreasonably
difficult. Maybe I'll tackle that after I get tired of hacking on
nsjsapi ...

[Jim Davidson]

Howdy,

I wrote the core SOB code years ago -- at the time I think our goals
included:

-- Security: Server connects out to known clients.
-- Performance: State-full connections, minimal protocol overhead,
lots of caching, no encoding overhead
-- Simple: String keys, string data, integration with net cache flush
-- Insight: Lot's of traffic counters collected
-- Tcl aware: Expects to store/send UTF-8 strings only
-- Some special case stuff, e.g., the comment board code.

In general, SOB met these goals at the expense of:

-- Config complexity: Maintaining topology is rationale but tedious
compared to client-server HTTP connections through routers
-- Binary: Just strings
-- File system scale: SOB relies on rationale topology and/or key
management to avoid file system scale problems (i.e., too many files
in a directory).

The point on UTF-8 deserves a bit more explanation. In general, all
the parts of the "dci" module assumed everything is in UTF-8,
always. In a sense, it's best to imagine SOB (and NV, AV, etc.) as
really just a way to serialize Tcl data to/from some stable something
-- if it's not in memory, it's designed to be read directly into
memory with no encoding / translation / etc. In other words, the
lack of encoding and binary code support was by design, not an
oversight, which provided some benefits but -- as in the case of
trying to use SOB for thumbnails -- some downside.

-Jim

[Damien O'Rourke]

Jim,

thanks for the info.
Keeping it all in utf-8 makes sense as you say ... so long as you
don't need to access binary files directly with another app. If all
send/receive requests go through the sob it works fine ... I have one
service using it that way at the moment. Debating about whether to
switch it to using nproxy ... there's some sense in making it the
gateway for all requests for those files ... there are other systems
that use similar models e.g. email de-duping software i.e. the black
box service ... who knows how YouTube or AWS actually store the
data ... could be own propriety format for all I can know ... all I
see is what they present to me when I request an item.

Damien

[Tomasz Kosiak]

On 10/30/07, Damien O'Rourke <doro...@carsales.com.au> wrote:
> Jim,
>
> thanks for the info.
> Keeping it all in utf-8 makes sense as you say ... so long as you
> don't need to access binary files directly with another app. If all
> send/receive requests go through the sob it works fine ... I have one
> service using it that way at the moment. Debating about whether to
> switch it to using nproxy ... there's some sense in making it the
> gateway for all requests for those files ... there are other systems
> that use similar models e.g. email de-duping software i.e. the black
> box service ... who knows how YouTube or AWS actually store the
> data ... could be own propriety format for all I can know ... all I
> see is what they present to me when I request an item.
>
> Damien

See Google Tech Talk video on YouTube Scalability

http://kylecordes.com/2007/07/12/youtube-scalability/

--tkosiak