Web transmission spec
Norman Gillaspie
There is a new batch type being uplinked that will be used for
uplinking files (mostly web pages).
The batch type is called FILE and will be sent in both normal (TEXT)
or compressed (GZIP) modes just like all the other batch types.
TEXT mode does not imply ASCII, just that it is not compressed.
The batch consists of multiple files, which are identified by
their URL and a byte count. There will be a line containing
the URL followed by whitespace and a bytecount, and terminated
with a CRLF. The next "bytecount" bytes of data are the file
contents. Another URL can then follow. Note that URLs can be
of any type (http, ftp, etc) and be from any host. If a web
page contains multiple graphic files we will attempt to
send them all within the same batch (but not guarranteed).
---------
So a FILE batch might be look like this:
batch type FILE, format GZIP:
The file is gunzipped and looks like this:
http://www.ncit.net/index.html 3432<CRLF>
3432 bytes of data
http://www.ncit.net/logo.gif 322<CRLF>
322 bytes of data
---------
An updated to the psnews program will save these new FILE type batches
to a separate directory for processing. A new program will be
distributed soon that will pull out the URLs and store them on a local
disk for viewing with a standard browser.
Currently we are transmitting certain URLs each hour. This will
eventually change to where things like a weather satellite photo
will only be sent when a new image is available.
The format of the FILE batch is subject to change for the next
few months until we have some experience with using it.
There is a possibility the URL identification lines will
be followed by other data (such as Last-Modified headers from
web servers) before the actual file data occurs, if things
like this are found to be useful. Suggestions to n...@pagesat.net
are welcome.
In addition if you wish to be notified with upcoming changes, maintainance,
schedule, the new satellie moves etc. Please send a message to n...@pagesat.net
requesting to be added to the mail list.
I have sent to messages so far. If you have not received them then
you should be added to the maillist.
Norman
Jack Kilday
>An updated to the psnews program will save these new FILE type batches
>to a separate directory for processing. A new program will be
>distributed soon that will pull out the URLs and store them on a local
>disk for viewing with a standard browser.
The companion program to which Norman refers is not yet available, but
if any of the DOS PSFile users are interested, an experimental version of
PSFile capable of saving these new FILE type batches to a separate directory
is available.
You may obtain via anonymous ftp from:
ftp.nlbbs.com: /pub/pagesat/ps_exp.zip
Comments welcome.
- Jack
--
Jack Kilday *==* jki...@nlis.net | Maine's Premier ISP & Oldest IBM BBS
*Northern Lights Internet Services* | * * * Telnet to 'nlbbs.com 6000' * * *
207-761-1991 * http://www.nlbbs.com | The Home of ZipNews *==* ftp.nlbbs.com
Stephen Lloyd
JK>From: jki...@server.nlbbs.com (Jack Kilday )
>Subject: Re: Web transmission spec
JK>The companion program to which Norman refers is not yet available, but
>if any of the DOS PSFile users are interested, an experimental version of
>PSFile capable of saving these new FILE type batches to a separate directory
>is available.
Will the new version allow FIDO files over 200K to be processed without
CRC errors, making them useless?
---
* QMPro 1.01 42-9494 * The Informal BBS 6096526182 1:2623/88
Chris Fourroux
Jack Kilday wrote:
>
> The companion program to which Norman refers is not yet available, but
And for those of you that can't wait:
/*
9/30/96 -Chris Fourroux (op...@opie.com)
Utility to extract multiple binary image files
from one text file of new NCIT "url" type.
Used Turbo C to compile and test on DOS system.
I tried to keep this as portable as I could, But I don't claim
to be a programmer.
*/
#include <string.h>
#include <stdio.h>
#include <stdarg.h>
int main(int argv,char *argc[])
{
FILE *stream,*out;
char msg[81],*ptr;
char string[4][80];
char fn[21];
unsigned long size=0,l=0;
int i;
char buf[2048];
if (argv<2){
printf("format \"GETIMAGE FILENAME\"\r\n");
return 1;
}
if ((stream = fopen(argc[1], "r+b"))
== NULL)
{
fprintf(stderr, "Cannot open input file.\n");
return 1;
}
while (!feof(stream)){
fgets(msg, 80, stream);
if (!feof(stream)){
sscanf(msg,"%s %lu", &string[1], &size);
if (strncmp(string[1],"http",4)!=0){
printf("Invalid input file\r\n");
fclose(stream);
return 1;
}
ptr = strrchr(string[1],'/');
sprintf(fn,"%s",&string[1][ptr-string[1]+1]);
if ((out=fopen(fn,"w+b"))== NULL){
fprintf(stderr, "Cannot open %s .\n",fn);
return 1;
}
i=0;
l=0;
while (l<size){
if ((l+2048)<size)
i=2048;
else
i=(size-l);
fread(buf,1,i,stream);
fwrite(buf,1, i,out);
l+=i;
}
fclose(out);
printf("%s\r\n",string[1]);
printf("%lu Bytes written\r\n",size);
}
}
fclose(stream);
return 0;
}
Chris Fourroux
Chris Fourroux wrote:
>
> Jack Kilday wrote:
> >
>
> > The companion program to which Norman refers is not yet available, but
>
> And for those of you that can't wait:
availible at ftp://opie.earthlink.net/getimage.c
also availible are some examples of the images
extracted from these new TEXT/url files and
a precompiled DOS executable of the above Source
code.
Chris Fourroux
> Jack Kilday wrote:
> >
> > The companion program to which Norman refers is not yet available, but
> And for those of you that can't wait:
I do , in fact, now claim to be a programmer.