D3 Web Server Project

[Rick Smereka]

I am near the end of a project writing a D3 web server. It serves static
and dynamic (BASIC) pages.

There are a couple of development issues that I would like to discuss. I
am getting this nagging feeling that there is a better way to implement
a couple of things.

The project name is W3. The dynamic pages have the extension 'w3b'.

The server has one D3 file of web pages and blob indexes called W3PAGE.

Static pages have the extension 'html' and are stored as-is in an item.
The item ID is the full path and name of the page (eg: /index.html).
This includes the path separators.

Blobs are not stored in D3. They are stored on the host operating system
disk and loaded at run time. The D3 blob item contains the path and file
name of the blob in the item ID and the operating system path and file
name in attribute one.

Dynamic pages are flash compiled BASIC modules (not cataloged) and
loaded dynamically at run time by the D3 'call @var()' construct. The
compiled module is stored in W3PAGE with the item ID as the full path
and file name of the web page.

I have plans to port W3 to other MV systems.

These design decisions seem to work well until I realized that it is
probably not a good idea to have forward slashes in the item ID. D3 does
not seem to mind this but it refuses to compile a BASIC module with a
slash in it's name (eg: SUBROUTINE /index.w3b()). When I create sample
dynamic pages, I eliminate the entire path and only use the module file
name and extension.

I also have concerns about how long (in characters) the item ID can be.
I realize that there is no practical limit in most MV environments but
many of those environments only consider a portion of the item ID as
significant (I think?). This situation may lead to multiple pages being
considered as having the same name. Since I am storing the full path and
page name in the item ID, there will be a lot of items that are long.

To compensate for the length of the page path/name, I was thinking of
having an index list in the W3PAGE file where each line in the item
would have a random series of characters followed by a space and the
page full path and file name. Web pages would then be stored in the
W3PAGE file with the item ID of the random series of characters.

Ideas anyone...
TIA
Rick Smereka

[Tony Gravagno]

> From: Rick Smereka

> I am near the end of a project writing a D3 web server. It serves
static
> and dynamic (BASIC) pages.

I'm curious why you decided to build the web server component into the
DBMS. A few of us did this with HTTP, SMTP, FTP, IRC, and other
protocols from 1995-2005,but then almost everyone stopped pursuing
that kind of development. Even Coyote, the most well-known MV web
server, was pretty much feature-complete by 1998 or so. It's raison
d'être was at an end when native environments like AP/Native and
AP/Pro went extinct and everyone was able to use "real" web servers
for all of their needs. And the NuWiki from Binary Star never really
took off either, given the plethora of free wiki platforms already in
existence. Building servers into MV platforms is cool and all, but
people spend years on other tools like that, like Apache, IIS, Casini,
lighthttpd, nginx, etc... I've learned that's usually better to
respect the work of others who specialize in these things, and
integrate with fine quality software, rather than trying to compete. I
wrote about this a few years ago here:
http://nebula-rnd.com/blog/tech/mv/2009/08/mv-to-anything.html

I'd be especially curious about your security and whether you can
support SSL. What about certs?
What about standard error return values (301, 404, 500, etc)?
What about Ajax and JSON, and handling different MIME types
(supporting REST or SOAP)?
And what about DoS attacks? What happens when a script kiddie hits
your host with a thousand simultaneous requests, or with one request
with a 20MB payload?
If you're using %sockets, are you aware of how sensitive D3 Windows is
when functionality as a socket server?
Do you support listening on any socket port or is it only 80?
What about session pooling? How are you handling load of two inbound
clients when one of them is doing a Select that takes 30 seconds, or
when one of them is returning a huge payload? Is every client request
processed by a different DBMS session=license?
Does the server support GET, POST, and DELETE separately?

Those are features provided by web servers. A web server doesn't just
serve up web pages. It supports security, concurrency, state, partial
page requests, and many other functions. I don't know how much time
you've invested in this but unless you can answer those questions
adequately then you'll need to put in more time - or change your
approach.

It's going to be tough if this is a commercial venture. D3 includes
free tools for web services now, which are easy to call from a common
(and free) web server. Of course that's different, but you'll need to
provide compelling value to compete with free. D3 also supports
FlashCONNECT.

> There are a couple of development issues that I would like to
discuss. I
> am getting this nagging feeling that there is a better way to
implement
> a couple of things.
>
> The project name is W3. The dynamic pages have the extension 'w3b'.

I immediately think of how everything in FlashCONNECT starts with 'w3'
but I guess that doesn't matter.

> The server has one D3 file of web pages and blob indexes called
> W3PAGE.
>
> Static pages have the extension 'html' and are stored as-is in an
item.
> The item ID is the full path and name of the page (eg: /index.html).
> This includes the path separators.

We've discussed here a few times the relative merits of using "real
data" as item keys, and the is consensus is that it's a bad idea.
Never use last names, phone numbers, SS#, or anything meaningful as an
ID, because undoubtedly you'll have a need to change it later. A
company that uses some other company's product key is up a creek when
that other company decides to revise their SKUs. And someone in the
company might decide it's a bad idea to send order IDs beginning with
T to Walmart because you're already using W to prefix all Walgreens
orders. Just don't do that - and don't use real page names as ID's
either. Do what they do in CMS packages. All resources get a unique
sequential ID. You can assign an alias to an ID as well as a category,
so path category/alias.html gets that ID. You can also create a table
that allows access to various aliases or IDs based on patterns. So
products/telephones/lg/p999 gets you to the same place as
/smartphones/lg999. You can also use this to setup 301 redirects since
you probably don't have .htaccess and to handle requests for
robots.txt, and other common files.

> Blobs are not stored in D3. They are stored on the host operating
> system disk and loaded at run time. The D3 blob item contains the
path
> and file name of the blob in the item ID and the operating system
path
> and file name in attribute one.
>
> Dynamic pages are flash compiled BASIC modules (not cataloged) and
> loaded dynamically at run time by the D3 'call @var()' construct.
The
> compiled module is stored in W3PAGE with the item ID as the full
path
> and file name of the web page.

I hope you validate requests against your W3PAGE dictionary before you
use call @var, otherwise it will abort right there. Call @() is also
very expensive because it needs to resolve dynamically. There are
other ways to do this.

> I have plans to port W3 to other MV systems.

I suspect you'll get either (a) much less feedback from people who
are already using common web technologies, or (b) the feedback will be
much more brutal than what I'm dishing out here. So be prepared...

> These design decisions seem to work well until I realized that it is
> probably not a good idea to have forward slashes in the item ID. D3
> does not seem to mind this but it refuses to compile a BASIC module
> with a slash in it's name (eg: SUBROUTINE /index.w3b()). When I
> create sample dynamic pages, I eliminate the entire path and only
use
> the module file name and extension.

Suggestion 1) don't use real page names as program or item IDs.
Suggestion 2) if you really need to make dynamic Calls, write a code
generator that creates a jump table from existing page names. So a
table translates page /smartphones/lg999 to program SMART1, and you
generate a program that says:
CASE PATH = "/smartphones/lg999" : CALL SMART1(params)

Of course that means all page updates need to have that program
regenerated, but that could be a small price to pay here. You might
say "for a large site that's not practical" and I'd say "a large site
would be using Apache or IIS".

> I also have concerns about how long (in characters) the item ID can
be.
> I realize that there is no practical limit in most MV environments
but
> many of those environments only consider a portion of the item ID as
> significant (I think?). This situation may lead to multiple pages
being
> considered as having the same name. Since I am storing the full path
> and page name in the item ID, there will be a lot of items that are
long.

Yeah, that's another problem with the scheme, and there will be more.
Don't try to solve these cascading problems, change the approach and
all of the problems disappear.

> To compensate for the length of the page path/name, I was thinking
of
> having an index list in the W3PAGE file where each line in the item
> would have a random series of characters followed by a space and the
> page full path and file name. Web pages would then be stored in the
> W3PAGE file with the item ID of the random series of characters.
>
> Ideas anyone...
> TIA
> Rick Smereka

That's where I was going above, but of course we both know the ID
isn't going to be "random". Rather than using a single cross-reference
item, I think a file with one or more items per page/resource would be
better. It will be easy for your single item to get seriously bloated
and you'll find yourself wanting to re-design that. This follows a
common example where old MV apps were written with a list of order
detail record IDs stored in an order header. As soon as you get a
thousand lines of detail it becomes apparent that this model doesn't
work. Or put a thousand open orders in a customer master and you'll
come to the same conclusion. No, single item's aren't good for that,
use a file of items where space isn't an issue and where you can
easily modify the structure.

HTH
Tony Gravagno
Nebula Research and Development
TG@ remove.pleaseNebula-RnD.com
http://Nebula-RnD.com/blog
Visit http://PickWiki.com! Contribute!
http://Twitter.com/TonyGravagno

[Symeon Breen]

I agree with what tony has said, but if you want to do this I would
seriously consider implementing routing. This will solve a lot of your path
issues and the storing of / in id's etc. Basically routing (made popular
because of REST and MVC) means your url structure needn't reflect your
document structure, so you can create routing profiles that both define code
and data to be used

e.g. you could define a route of
/site/{programfile}/{programname}/{data1}/{data2}

so if I came to www.mysite.com/site/WBP/webProg1/hello world/7 it
would call webProg1 in the file WBP and pass parameters "hello world" and 7


take a read about the asp.net routing as it is fairly fully featured.

--
--
You received this message because you are subscribed to the "Pick and
MultiValue Databases" group.
To post, email to: mvd...@googlegroups.com To unsubscribe, email to:
mvdbms+un...@googlegroups.com
For more options, visit http://groups.google.com/group/mvdbms

---
You received this message because you are subscribed to the Google Groups
"Pick and MultiValue Databases" group.
To unsubscribe from this group, send email to
mvdbms+un...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.



-----
No virus found in this message.
Checked by AVG - www.avg.com
Version: 2012.0.2221 / Virus Database: 2639/5561 - Release Date: 01/27/13

[rsme...@future-lab.com]

Thanks for your candid response Tony and Symeon,

>I'm curious why you decided to build the web server component into the
>DBMS.

There are a whole bunch of reasons: Having the data, language and web
server together without going through all those middle-ware contortions;
Being able to use BASIC as a scripting language; Having the security of
an alien HTTP server that does not identify itself in any way and the
crackers do not understand; Running a server that I build and completly
understand; Working through the HTTP protocol; and there are more.

>A few of us did this with HTTP, SMTP, FTP, IRC, and other
>protocols from 1995-2005,but then almost everyone stopped pursuing
>that kind of development. Even Coyote, the most well-known MV web
>server, was pretty much feature-complete by 1998 or so.

I am well aware of the history of implementing web protocols with various
MV environments and I am not encumbered by what other people think or are
developing.

>support SSL

No

>certs?

No

>What about standard error return values (301, 404, 500, etc)?

Support for 200, 302, 404, 505, 500

>What about Ajax and JSON, and handling different MIME types
>(supporting REST or SOAP)?

Different mime types are supported and can be changed at run time.

>And what about DoS attacks? What happens when a script kiddie hits
>your host with a thousand simultaneous requests, or with one request
>with a 20MB payload?

The server does not support uploading data files so they cannot send that
payload and the server would never try to return a 20mb payload even if
they knew how to trigger it. I have a receive limit on the server which,
when exceeded, the connection is dumped.

>If you're using %sockets, are you aware of how sensitive D3 Windows is
>when functionality as a socket server?

Yes. One of my document references is:
Tigerlogic TA D3/NT 7.2.1 Socket Communications on D3/NT
http://forums.tigerlogic.com/index.php?showtopic=30

>Do you support listening on any socket port

Yes

>What about session pooling?

Right now, there is none. Single session. According to D3 doc's, the
'%accept' function has a queue depth of 9 connections waiting.

This is another topic that I was going to bring up although another one of
my reference docs is:
Get D3 days onto your web site using JD3 and Php
http://www.djpatterson.com/jd3php.html

I was going to implement session pooling similar to what JD3 has done
although this is still a fixed (not dynamic) solution. I have already
planned for multiple process routines.

>Is every client request
>processed by a different DBMS session=license?

No. One license for the whole server. Right now, it it not a phantom, I
just launch it at TCL.

>Does the server support GET, POST, and DELETE separately?

GET, POST and HEAD are fully supported.

>It's going to be tough if this is a commercial venture.

Not commercial. Like everything I produce (save company projects), W3 is
licensed GPL V2.

>We've discussed here a few times the relative merits of using "real
>data" as item keys, and the is consensus is that it's a bad idea.

Agreed.

>I hope you validate requests against your W3PAGE dictionary before you
>use call @var, otherwise it will abort right there.

Absolutely. I look in the dictionary for the object code assuming that if
the object code is there, the module will execute.

>Rather than using a single cross-reference
>item, I think a file with one or more items per page/resource would be
>better. It will be easy for your single item to get seriously bloated
>and you'll find yourself wanting to re-design that.

I'll consider that.

>if you want to do this I would
>seriously consider implementing routing.

I don't see the need for that.

>take a read about the asp.net routing as it is fairly fully featured

Asp.net... Not even at gun point.

Rick Smereka

[Tony Gravagno]

> >if you want to do this I would
> >seriously consider implementing routing.
>
> I don't see the need for that.
>
> >take a read about the asp.net routing as it is fairly fully
featured
>
> Asp.net... Not even at gun point.
>
> Rick Smereka

Rick, thanks for your responses to my brain dump of functionality.
None of this is intended to be critical, but you've written some
serious software which requires serious inquiry.

Symeon was right. Try not to get religious about technology. If you're
creating a tool which is supposed to implement common functionality,
as you've demonstrated, you need to look around for examples. Forget
ASP.NET and think about mod_rewrite. Like PHP? Look at routing in
CodeIgniter or Zend. If you're familiar with Java, see Stripes or
Spring or the UrlRewriteFilter for Tomcat. What about Selector for
Python or Routes for Ruby? It's all the same.

I'm glad you read the notes about %sockets - I'm surprised that wasn't
discouraging for this specific application. One of the reasons why I
got away from sockets some years ago is that the code simply isn't
portable. Coding that far into the platform doesn't earn commensurate
returns. And porting to a different platform is so painful that that
it's just not worth it to try to support multiple platforms.

Kudos on FOSSing it, but as many will tell you here, there aren't many
people who will contribute to FOSS in this industry, so that concept
doesn't carry as much weight as it does in other arenas.

Have you seen this effort?
http://sourceforge.net/projects/mvwww/
That's a FOSS alternative to FlashCONNECT.
Registered 2004, last updated 2009, cross-platform, zero industry
interest.

Aside from being cool, anyone using this needs to ask themselves why
they aren't using a standard web server when these days it's so easy
to do so. Hell, even people who hung on for a decade without using a
browser are now talking about how they're coding their web interfaces.
For business use, I can't see an IT manager trying to explain this
choice to management, especially when it's GPL, which these days means
"someone might get to fixing security issues when there's some free
time".

Except for being single-threaded, this MV-based web server sounds
cool, but just a bit too little and a bit too late. This sounds like
an interesting project for hobbyists, maybe as a learning tool. That's
where I see it anyway. Others can decide for themselves.

Sorry for the badgering, I'm just hammering out the thoughts so I can
get back into code. Thanks for your patience.
T

[Dick Thiot]

Tony and Symeon,

Because Rick is writing a MV based web server doesn't mean that he is writing a general purpose web server that is designed to service the needs of others.  It sounds to me like he is writing a specialized web server to interface with his particular needs.  I can see security advantages to this sort of approach.

Rick, 

I would suggest that you run some of the existing external test processes to determine security as well as ability to handle the load that you expect.

Sorry for top posting but since these were general comments rather than specific to other replies I just decided to put it here and besides, it's just easier when you are up in the middle of the night with a head cold.  No sympathies, please! ;-)  At least that is my excuse for being lazy right now and I am sticking to it!

Dick

T

--
--
You received this message because you are subscribed to
the "Pick and MultiValue Databases" group.
To post, email to: mvd...@googlegroups.com
To unsubscribe, email to: mvdbms+un...@googlegroups.com
For more options, visit http://groups.google.com/group/mvdbms

---
You received this message because you are subscribed to the Google Groups "Pick and MultiValue Databases" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mvdbms+un...@googlegroups.com.

[rsme...@future-lab.com]

>Please do everyone a favor and if you're responding to a digest then
>please change the subject to something appropriate.

Sorry about that. I forgot to change the subject like.

>http://www.brianleach.co.uk/pages/mvscript.htm
>Go to the "Live Demo" page, then click "download the source", and look
>at the .wsp files.
>I predict your jaw is going to drop.
>Of course that's a commercial offering, and only available for U2, but
>it's also well-supported.

Looks pretty much like W3 source except he uses the script trigger '<%' and
'%>' and he
does 'Crt' to do raw output.

>How about MOST open source projects? ;b

Can't say. Like everything else, some good, some bad.

r