Cannot get VBS to retrieve a webpage using HTTPS ..

[R.Wieser]

Hello all,

Does anyone know if there is a MS Active-X component which will allow me to
retrieve a HTTPS webpage ? I've tried Microsoft.XMLHTTP as well as
Msxml2.ServerXMLHTTP.6.0 , but both return a 0x800C0008 error: "The download
of the specified resource has failed" (the used URL was
"https://google.com/index.html", which returns a(n acceptable) 404 when HTTP
is used.

Regards,
Rudy Wieser

[Mayayana]

"R.Wieser" <add...@not.available> wrote

| Does anyone know if there is a MS Active-X component which will allow me
to
| retrieve a HTTPS webpage ?

This seems to work, but I don't normally use it, so I'm
not familiar:

Set WinHttpReq = CreateObject("WinHttp.WinHttpRequest.5.1")

temp = WinHttpReq.Open("GET", "https://www.washingtonpost.com/", True)
WinHttpReq.Send

WinHttpReq.WaitForResponse

strResult = WinHttpReq.ResponseText


Set FSO = CreateObject("Scripting.FileSystemObject")
Set TS = FSO.CreateTextFile("C:\windows\desktop\up\index.html", True)
TS.Write strResult
TS.Close
Set TS = Nothing

Set fso = Nothing
Set WinHttpReq = Nothing

MsgBox Err.number & vbCrLf & Err.description
'------------------------------------------------

google.com gave me a 301
www.google.com worked but gave me a 404.
I then tried https://www.washingtonpost.com/
and got a cert expired error.

[R.Wieser]

Mayayana,

> This seems to work, but I don't normally use it, so I'm
> not familiar:
>
> Set WinHttpReq = CreateObject("WinHttp.WinHttpRequest.5.1")

Thank you ! I quick test shows that with it I can now retrieve pages from
websites which earlier "refused" to cooperate. :-)

Now I wonder how many different "retrieve a webpage" Active-X objects are
actually on my 'puter ...

Regards,
Rudy Wieser


-- Origional message:
"Mayayana" <maya...@invalid.nospam> wrote in message
news:orl5b4$7qk$1...@gioia.aioe.org...

[R.Wieser]

> I quick test shows that with it I can now retrieve pages from websites
> which earlier "refused" to cooperate. :-)

I can get the website to respond, but for some reason the content is
delivered as raw GZIP compressed data (even when I set the accept-encoding
to "identity").

In other words: One problem down, but another pops up; :-( :-)

Anyone know how to tell the WinHttp.WinHttpRequest.5.1 to reverse any
applied transport encoding/compression ?

Regards,
Rudy Wieser


-- Origional message:

"R.Wieser" <add...@not.available> wrote in message
news:orleaj$mvv$1...@gioia.aioe.org...

[Mayayana]

"R.Wieser" <add...@not.available> wrote

| I can get the website to respond, but for some reason the content is
| delivered as raw GZIP compressed data (even when I set the accept-encoding
| to "identity").
|

Identity means uncompressed? The RFC doesn't seem to
explain that. I wonder if it might work to put something
like "applepie", thus causing the server to conclude that
it doesn't offer anything you can use.

If you don't get it, I can give you VB6 code for GZip.
I also have components, sjHTTP and jsHTTP2, to deal
with it.
www.jsware.net/scripts.php5

Or you might be able to find other options online.
It's common and fairly simple encoding.

[R.Wieser]

Mayayana,

> Identity means uncompressed?

Among others, yes (in whole: Don't touch the contents. No encoding, no
translation)

> If you don't get it, I can give you VB6 code for GZip.

Thanks. But I've got zlib1.dll here, and enough knowledge to build me
another Active-X component if must be. :-)


But ... I've been cought out by one of the oldest "gotcha!"s there is: I did
set the accept-encoding, but a few lines down I had another such setting
which added the gzip encoding to it. <embarrassed>

Once I noticed and disabled it, I got the data I wanted/needed back. In
short: problem solved (unless a server refuses to send data non-gzip-ed
ofcourse).

But I'm wondering: when is getting your hands on raw
transport-compressed/encoded data usefull ? Can't really think of a
situation, or it must be a rather contrived one.

Regards,
Rudy Wieser


-- Origional message:

"Mayayana" <maya...@invalid.nospam> wrote in message

news:orlhhv$sda$1...@gioia.aioe.org...

[Mayayana]

"R.Wieser" <add...@not.available> wrote

| But I'm wondering: when is getting your hands on raw
| transport-compressed/encoded data usefull ? Can't really think of a
| situation, or it must be a rather contrived one.
|

I don't understand the question. You seem to
be conflating encryption with compression. The
former is for security, as I'm sure you know. The
latter is only a courtesy to the server, minimizing
the bytes they have to transfer. Basically saying,
"I can take a smaller package and unpack it on my
end, if that's easier for you."

(Though in these days of 1 MB worth of javascript
and 5 MB worth of video ads in a webpage, I don't
feel very sympathetic about server load on commercial
pages.)

[R.Wieser]

Mayayana,

> I don't understand the question. You seem to
> be conflating encryption with compression.

Whut ? Aren't those just two words for the same thing ? <drool> :-p

But no, not really. Although sometimes the only difference between the two
seems to be a secret key. :-)


The question is, what can I/someone *do* with data thats still compressesed
with (something like) gzip ? Its rather useless in that form, don't you
think ?

The only thing I can think of is when the program itself does not (want to)
use the data, but only acts as a go-between (think gateway proxy. It makes
no sense to decompres only to directly after it have to recompress again),
and/or to save it for a next request of the same data (a buffering proxy).

Than again, when being in such a position it would be rather logical to also
put some kind of virus/malware/whatever scanner there, which would ofcourse
mean the data also needs to be decompressed.

So, I'm wondering why WinHttp.WinHttpRequest.5.1 returns raw, still
compressed data when the usefullness of it seems to be nil, and *always*
needs to be handled by yet another (non-existing in VBS as far as I know)
object ? It only seems to serve to increase the complexity of the program
(no matter which language its written in) ...

Regards,
Rudy Wieser


-- Origional message:
"Mayayana" <maya...@invalid.nospam> wrote in message

news:orlpdh$19se$1...@gioia.aioe.org...

[Mayayana]

"R.Wieser" <add...@not.available> wrote

| The question is, what can I/someone *do* with data thats still
compressesed
| with (something like) gzip ? Its rather useless in that form, don't you
| think ?
|

The data is usually being collected by a browser.
If you write your own page fetcher you should
add gzip support. It's just part of the job of
downloading webpages. All of that is normally
hidden by the browser.

I think winhttp is meant to be a clean, barebones,
flexible option. The easy API functions, like
URLDownloadtoFile, are just IE wrappers that can
affect cache and cookies. Likewise, IE automation
is sloppy and indirect. But winhttp is clean. It's just
a light wrapper for winsock, with conveniences.

I said before that I don't use it but I actually forgot.
I don't use it in script, but I use it in software. I have
a program that calls Google map server to get maps,
satellite, streetview, etc. I used to do that from
scratch with winsock. The only real options were winsock
on one extreme or IE wrappers on the other. IE wrappers
are hokey. Winsock is a lot of work and can't hand;e
encryption. But since XP, wnhttp is pre-installed
on Windows and I discovered that it would allow me
almost full access to the transaction while still taking
care of things like encryption, server handshake, etc.

[R.Wieser]

Mayayana,

> If you write your own page fetcher you should
> add gzip support. It's just part of the job of
> downloading webpages

Thats just it The object is written to specifically cater to HTTP
transport -- it has method for adding headers, and will, on reception, split
headers and content -- but for some reason doesn't (fully) revert the (to
me) transport-only changes made to the data (gzip, deflate and/or others).

So in a sense I agree with you (its part of the job of downloading
webpages), but I get the distinct feeling we still are not on the same line
here ...

> I think winhttp is meant to be a clean, barebones, flexible option.

I think you're right in the "barebone" approach. I'm just consider the
absense of reverting the (as far as I'm concerened) the transport
compression as "going to the marrow". :-(

> But since XP, wnhttp is pre-installed
> on Windows and I discovered that it would allow me
> almost full access to the transaction while still taking
> care of things like encryption, server handshake, etc.

That might be. But have you *ever* wanted to do, or even *thought* of doing
your own compression ?

And besides if you did, an object like this which would handle the more
known compression methods would still allow you to do so (you would just
need to come up with your own "accept-encoding" compression name -- or have
it include a "give me the raw, un-decompressed data" method).

No, it doesn't really make sense to me -- no matter in which language you're
using the object -- not to include support for the known types of
compression. You always use them.

> I used to do that from scratch with winsock. The only real options
> were winsock on one extreme or IE wrappers on the other. IE
> wrappers are hokey.

I've done the same myself. And yes, those IE wrappers are ...
interresting. You can disable quite bit of its functionality though,
including caching and cookie handling.

> Winsock is a lot of work and can't hand;e encryption.

Yeah, I've run into the same thing myself. And although I found some
listings showing how to create an SSL connection using MS DLLs only, I never
found (msdn) information on the dll functions themselves (which means I
had/have to try to guess the usage of quite a few arguments and results) :-(

Oh well. Who said that programming would be easy, just stringing functions
together ? If they did they probably have never dealt with MS. :-)

Regards,
Rudy Wieser


-- Origional message:
"Mayayana" <maya...@invalid.nospam> wrote in message

news:orm3fl$1qfj$1...@gioia.aioe.org...

[Mayayana]

"R.Wieser" <add...@not.available> wrote

| No, it doesn't really make sense to me -- no matter in which language
you're
| using the object -- not to include support for the known types of
| compression. You always use them.
|

I agree, but it's also not common for people
to write their own downloader. For me personally
winhttp is perfect because I have the zlib code
and it's not a problem to add decompression
functionality. Encryption, on the other hand, is
a headache. I spent a lot of time awhile back
trying to write my own encryption routines for
sending email, using one of a number of options.
I couldn't get TLS to work, couldn't find clear
docs, and couldn't find anyone who seemed to
know what they were talking about. (I haven't
even looked at http encryption.)

As with a lot of things, people offer code and help
without really knowing the topic. Lots of people
had samples for sending email through gmail using
SSL/TLS starting on an open port. No one had
samples for using STARTTLS on port 465. The former
negotiates a transition to encryption. The latter
starts encrypted. Most people didn't understand
the difference. ("Why do you need to use anything
other than gmail?") I finally gave up. So it was a big
relief to find that winhttp could wrap encryption
for me.

I've noticed a similar pattern whenever things are
difficult. At one point I wrote VB6 code to handle the
CDECL calls in cabinet.dll, so that I could work with
CAB files. VB experts had been offering CAB code for
years. VB Programmers Journal even had an article
about it once. In all cases the code was calling
SetupIterateCabinet from setupapi.dll, because VB
could make that call easily. But that could only access
one of 3 types of CAB file! So it was essentially useless.
But no one seemed to know that, including the VB6
experts.

You may already have ZLib code. If not and you
want to add that to your project, I have a sample
here:

www.jsware.net/jsware/vbcode.php5#zlib

It's easy to use, but takes a little setup because
it deals with byte strings. You can't just hand zlib
a file.

[R.Wieser]

Mayayana,

> I agree, but it's also not common for people to write their
> own downloader.

Yep. Most people just want "something which works". I myself am rather
prone to wanting to know *how* something works, and that includes trying to
write such stuff myself. Like, it seems, you.

> Encryption, on the other hand, is a headache. I spent a lot of
> time awhile back trying to write my own encryption routines
> for sending email,

I would not even try to do that. I'm no maths wizz, and as such stay far
away from it. But if you mean if I tried to use Keberos and other OS
available methods ? Yep. With stressing the "tried". As with you it
turned out to be *very* hard to find applicable documentation. MSDN is
nice, but rather often it doesn't actually help me advance codewise.

> As with a lot of things, people offer code and help without really
> knowing the topic. Lots of people had samples for sending email
> through gmail using SSL/TLS starting on an open port.

You have ? Hopefully not using OpenSSL, as that one, as far as I'm
concerned, sucks donkyballs. Too many methods to open a socket, and even a
rather usefull function which dependantly caused my program to crash. And
after a lot of hair-pulling and some googeling it was even *documented* (of
sorts) to do so under Windows. :-\

> No one had samples for using STARTTLS on port 465.

In such a cases I always try to spy on the communication of a program which
already can do it (possibly using wireshark or something like that).

I didn't even come that far though. After I found out that *anyone* could
send email under my name anyway (even dropped on my own ISP's email
servers!) I concluded that trying to create an SSL connection to send my
mail thru simply wasn't worth it.

> I've noticed a similar pattern whenever things are difficult. [snip]

Thats sounds awfull and remarkable. Though I cannot imagine nobody has
ever tackled the other two ...

My own story in that regard is that I, years ago, spend a *lot* of time with
decoding the .DBX files belonging to Outlook Express. Found all kinds of
data on the file format and eventually succeeded in listing its contents
(and toggle some flags). But what I do not remember *ever* having found was
examples using directdb.dll, which makes the whole thing a *lot* easier (and
allows you to access those .DBX files even when outlook express is running).
Even now a quick peek shows that the first 5(!) google result pages have
nothing on it.

And than ofcourse the bad encounters: Like a few months back me posting a
simple question with all the relevant data and a code snippet in a C++
forum, and than getting flack because I either should not even be thinking
of doing something like that (thats for me to decide, thankyouverymuch),
because I didn't give data such-or-so (at which point I refer them back to
my first post which contains it, after which ofcourse a lot of bickering
ensues), or demands for the rest of the code so they can figure out what I'm
*really* after (as I'm asking I simply *must* be a noob), and than work
around the posted problem by throwing something fully different at it.
Quite a self-righteous lot, often not having a clue to what the answer is
(or even might be), but thinking that with enough bullshitting around it I
will not notice that. :-((

> You may already have ZLib code.

I do, and have succesfully transposed a number of such code examples to my
own favorite language (Assembly). But thanks. I (almost) never turn down a
link to a page with possibly interresting stuff like that (even though I've
not written any VB since a decade-and-a-half ago ...)

> You can't just hand zlib a file.

You .. can't ... ? :-p "(un)compress" comes quite close to it though.

Regards,
Rudy Wieser


-- Origional message:
"Mayayana" <maya...@invalid.nospam> wrote in message

news:orntk0$kae$1...@gioia.aioe.org...

[Mayayana]

"R.Wieser" <add...@not.available> wrote

|
| My own story in that regard is that I, years ago, spend a *lot* of time
with
| decoding the .DBX files belonging to Outlook Express.

I'd be interested in that if you have code or docs
you're willing to share. Just last week I wanted to find an
old email and went to an OE backup. I was surprised
to find that DBX extractors are very expensive. I
ended up with a search in Notepad. But if I had docs
I'd like to write a script to extract the emails into
a nice neat TXT file, or maybe an MSI database,
for better longterm backup.

|
| And than of course the bad encounters: Like a few months back me posting

a
| simple question with all the relevant data and a code snippet in a C++
| forum, and than getting flack because I either should not even be thinking
| of doing something like that
|

It's surprising how common that is. I get it all
the time, still, in the VB group. I've had arguments
over it. I think it's an issue of personality style.
Some people just really like rules and like nothing
more than to tell you their version of the rules.

The IE wrapper API was actually one of
those arguments. The commercial MS partner
types have a lot of ideas about what's kosher
and what isn't. They don't think anyone should
be mucking around with winsock. I pointed out
that the IE wrapper functions are rather hokey
and allow cache/cookies. Someone argued that
those are system cookies and cache, not IE
cookies and cache. As such, they're normal and
are supposed to be connected to downloads.
It was an odd debate. This was a professional
programmer with a lot of experience, but who had
completely fallen for Microsoft's IE-is-the-Internet
scam. There was no way to even explain the idea
that cookies and cache belong to browsers, not OSs.

I've never worked in any kind of tech office and
have been self-employed for most of my life, but
from what I've seen I get the sense that many
such operations -- like many office jobs in general
-- are paramilitary social structures with lots of
hierarchy. The kind of jobs where they're looking
for "dedicated team players" AKA lackeys who don't
mind working late for a set salary.

[R.Wieser]

Mayayana,

> I'd be interested in that if you have code or docs you're willing to
> share

I've got both. I can only verify its usefullness for OE as provided with
W98 and XP though.

I've got stored HTML pages (some from the now defunct oe5dbx.aroh.de
website -- if you want you could also check if the "wayback machine" still
has them -- and some ZIP files with downloads on the same subject in both
cpp and pascal.

I've got a full program with which you can browse messages inside a .DBX .
Its fully assembly though. And, don't laugh, I still got code written in
QBASIC which can do pretty-much the same -- though not as neat.

And I also have some unfinished(!) work on that directdb.dll I mentioned in
my previous message.

So, make your choice(s). And although I could copy/paste the textified
contents of the HTML pages and/or a listings in a number of messages here, I
don't think that is what this newgroup is ment for. In other words, I would
need some website to upload them to (plain HTML, otherwise my browser
refuses to cooperate -- on my strict orders ofcourse :-) ), or some
(temporary) email address. Or ofcourse a (binary) newsgroup ment for such
exchanges (of which I have no idea if they even exist, but we *could* use
a(ny) now-defunct group for it if need be).

> But if I had docs I'd like to write a script to extract the emails into a
> nice neat TXT file

Now you're forcing my hand ... I even wrote an ActiveX component (which I
access using VBScript) which is able, in its own clumsy way, to extract
whatever fields you want from it. But although it works, its currently not
really "user friendly" though.

> It's surprising how common that is.

Yup.

> Some people just really like rules and like nothing
> more than to tell you their version of the rules.

Tell me about it. Those, and the posting/usenet-rule nazis. They would
not recognise a top posting if it would punch them in the face, but
according to a number of them my posting style certainly is -- probably
because I always leave the message I'm replying to as an "attachement" at
the bottom. Always clearly marked as such, but some people simply cannot, or
just refuse to read. :-(

And when, after they categorically refuse to even consider that top posting
is a bit different, I than explain to them that a rule without a means to
enforce it is meaningless their mental needle just jumps a groove back and
they regurgite the same thing all over again.

> The commercial MS partner types have a lot of ideas about what's
> kosher and what isn't.

Yup. Met them too. I always wonder how they can actually write
*anything* new, as they seemingly are only allowed to use what MS has
already described (in docs or in example code presumably). And my
preferred "see how stuff breaks and learn from it" approach ? Its as if
they see water burn/satan personified. :-)

> I pointed out that the IE wrapper functions are rather hokey and allow
> cache/cookies. Someone argued that those are system cookies and cache,
> not IE cookies and cache.

Yeah, and where are those system cookies and cache supposed to be stored ?
Besides, IE *is* the system -- with the browser just being a rather small
graphical shell ontop of all the components which are part of the system.

Point-in-fact: being able to access the web by entering an URL in the
addres-bar of your *file* explorer. Being able to initiate a search by
putting two-or-more words in that same addres bar. All *without* having IE
installed (it annoys me to no end, with or without IE being installed).

> As such, they're normal and are supposed to be connected to downloads.
> It was an odd debate.

You can say that again. What a pompous moron. But yes, I met those types
too. I'm starting to learn to terminate conversations with them though.
Maybe I will get the hang of it before I'm old-and-dead. :-)

> The kind of jobs where they're looking for "dedicated team players"
> AKA lackeys who don't mind working late for a set salary.

Yup. But isn't that true of most jobs ? Although a humorously ment
website like https://notalwaysright.com/ is'n really representative, the
stories are often similar: Work till you drop, and don't you dare to give
lip.

Regards,
Rudy Wieser


-- Origional message:
"Mayayana" <maya...@invalid.nospam> wrote in message

news:oroblr$1f5k$1...@gioia.aioe.org...

[Mayayana]

"R.Wieser" <add...@not.available> wrote

| I've got stored HTML pages (some from the now defunct oe5dbx.aroh.de
| website

I got that. I also got the undbx code. But neither
provides any clear docs. What I was hoping for would
be either a typelib for directdb.dll or a clear explanation
of the file format so that I can parse it directly. The
oedbx package has some of that, but it's partially
guesswork and he doesn't put it all together.

|
| And I also have some unfinished(!) work on that directdb.dll I mentioned
in
| my previous message.
|

If you think that might be useful you could send
it, or anything else that looks promising, to
j s w a r e at j s w a r e dot net

It would be worthwhile just to thwart the clowns
who are trying to get $60 for a DBX extractor
program.

[Mayayana]

"Mayayana" <maya...@invalid.nospam> wrote

|| And I also have some unfinished(!) work on that directdb.dll I mentioned
| in
|| my previous message.
||
|

I just managed to build a directdb.tlb
out of directdb.idl. so now I have the object
model for use in VB. But I don't have docs
for it. The objects, like idatabase, are only
documented at MS to say "don't use them".
I can't find the same docs in any SDK help files.

[Mayayana]

More followup, though I'm not sure this is really
of interest to much of anyone. Undbx is an OSS
program that nicely extracts emails from a DBX and
writes them as separate .eml files. I may just use
that or I may figure out the code. Either way, I
guess I have my backup. Maybe I'll put my effort into
writing an HTA as a front-end to an MSI storage file,
as a way to back up old email and make it easily
searchable. That would probably be time better
spent than trying to write DBX code.

[R.Wieser]

Mayayana,

> But I don't have docs for it.

Although I've found some odds-and-ends docs (cpp and h files mainly) the the
only descriptive info I found is from MS itself:
https://msdn.microsoft.com/en-us/library/bb931215(v=vs.85).aspx But as
mostly with those, it doesn't help to understand what needs to be done.

I tackled that by writing an intercepting directdb dll , which allowed me to
look at what exactly was called (by outlook express 6), and what the used
arguments looked like. Quite enlightning. :-)

> The objects, like idatabase, are only documented at MS to say "don't use
> them".

I stopped caring about that when I discovered how usefull the "by ordinal
only" (don't touch, you should not even *look* at them!) functions in quite
a number of their DLLs are (thanks to Geoff Chapell for documenting quite a
number of them).

> What I was hoping for would be .... a clear explanation of the file

> format so that I can parse it directly.

Althougth that is what my program currently does (parse it directly), I
never tried to write any kind of documentation for it (which, mind you,
would probably just link all the tidbits of found info together).

> If you think that might be useful you could send it, or anything else

> that looks promising, to [snip]

I'll send you the source code of the program I tested the decoding of the
dbx (before using it in the stand-alone and ActiveX. Less crud to look
at ), as well as the program which I'm using to test directdb access with.

If you want (let me know) I can also send you all the gathered docs and zips
I found (not really much), but I get the feeling you already found those
yourself.

> It would be worthwhile just to thwart the clowns who are trying to get $60
> for a DBX extractor program.

:-) During my searches I encountered several such programs, and had the
same feeling. But as you seem to be a professional software writer your
time is worth quite a bit of money, and you will probably (and most likely
already have) spend enough time on it to dwarf that price point. :-p

I myself am just a hobby programmer, having no problem with spending (quite
some) time while trying to recreate something that already exists (and could
actually have costed (development time) and be worth its money) just to see
if I can. To me its either that, or reading a book.

Regards,
Rudy Wieser


-- Origional message:
"Mayayana" <maya...@invalid.nospam> wrote in message

news:orr0hj$1kaa$1...@gioia.aioe.org...