Could a unique number be determined for any webpage?
Lao Ming
would it be possible to obtain such a number such that any time I ran
my script, it would always determine a unique number for that very
page (as long as the web page exists)? If so, how could this be done?
I don't recall the math used but there used to be a way to get unique
decimal value for an IP address (it used to be calculated by MacTCP)
but it doesn't seem possible to obtain a value for any filepath. If
anyone has any ideas about this, I'd really appreciate it. Thanks.
Seebs
Do you want it for the page, or for the contents of the page?
In short, do you want to get the same value for 'http://www.cnn.com/' every
day, or only the same value for the next five or ten seconds before they
change something?
-s
--
Copyright 2009, all wrongs reversed. Peter Seebach / usenet...@seebs.net
http://www.seebs.net/log/ <-- lawsuits, religion, and funny pictures
http://en.wikipedia.org/wiki/Fair_Game_(Scientology) <-- get educated!
Lao Ming
> On 2009-11-17, Lao Ming <laoming...@gmail.com> wrote:
>
> > Suppose I want to assign a unique number to any website anywhere --
> > would it be possible to obtain such a number such that any time I ran
> > my script, it would always determine a unique number for that very
> > page (as long as the web page exists)? If so, how could this be done?
> > I don't recall the math used but there used to be a way to get unique
> > decimal value for an IP address (it used to be calculated by MacTCP)
> > but it doesn't seem possible to obtain a value for any filepath. If
> > anyone has any ideas about this, I'd really appreciate it. Thanks.
>
> Do you want it for the page, or for the contents of the page?
>
> In short, do you want to get the same value for 'http://www.cnn.com/'every
> day, or only the same value for the next five or ten seconds before they
> change something?
If the page is modified, it still has the same URL. It is the actual
unique URL that I want to put a decimal value on and not the
contents. Thanks.
> -s
> --
> Copyright 2009, all wrongs reversed. Peter Seebach / usenet-nos...@seebs.nethttp://www.seebs.net/log/<-- lawsuits, religion, and funny pictureshttp://en.wikipedia.org/wiki/Fair_Game_(Scientology) <-- get educated!
Ben Finney
> If the page is modified, it still has the same URL. It is the actual
> unique URL that I want to put a decimal value on and not the contents.
> Thanks.
In that case, simply generate a cryptographic hash of the URL and use
the resulting number for the unique identifier. An SHA-1 hash returns a
number, and should be adequate for what you've described so far.
--
\ “A free society is one where it is safe to be unpopular.” |
`\ —Adlai Ewing Stevenson |
_o__) |
Ben Finney
Seebs
> unique URL that I want to put a decimal value on and not the
> contents. Thanks.
Then, yes, a unique number could be determined for any web page.
-s
--
Seebs
> unique URL that I want to put a decimal value on and not the
> contents. Thanks.
Then, yes, a unique number could be determined for any web page.
-s
--
Lao Ming
wrote:
Thanks. It's perfect! I don't know why I didn't think of that. I
guess I have always just used md5 with filenames not thinking it could
be used with any string. I don't have sha1sum but I could use:
echo http://www.cnn.com |md5
Thanks again!
Seebs
>
> echo http://www.cnn.com |md5
>
> Thanks again!
You should consider, though: While it's exceedingly unlikely that you'll
encounter key clashes often, it is obviously the case that key clashes are
possible.
If you really want to be totally sure that you have a unique string, consider
treating the URL string as a base-whatever number, and just encoding the whole
thing. This absolutely guarantees uniqueness.
-s
--
jellybean stonerfish
An ip4 number is four 8bit values.
For an ip of 12.34.64.134
The decimal value is 12x2^24 + 34x2^16 + 64x2^8 + 134
Stephane CHAZELAS
[...]
> Thanks. It's perfect! I don't know why I didn't think of that. I
> guess I have always just used md5 with filenames not thinking it could
> be used with any string. I don't have sha1sum but I could use:
>
> echo http://www.cnn.com |md5
You may want to canonify the URL beforehand as
http://www.cnn.com
http://www.cnn.com/
http://www.cnn.com:80/
http://www.CNN.com:80/
Are all the same URL but have different MD5 sums.
http://www.CNN.com:80/#some-fragment
also points to the same page.
http://www.cnn.com//
http://www.cnn.com/.
Also generally do.
See also
http://www.cnn.com/WORLD
vs
http://www.cnn.com/WOR%4CD
and " " vs "+" vs "%20".
perl (at least) has some modules for URI handling (and also for
MD5 or SHA1 hashes)..
Also note that in that command line above, you're including a
trailing "\n" character in the string to hash.
--
Stï¿œphane
bb
you can also create a tinyurl for each url and make that to a number some way.
http://tinyurl.com/
/bb
Lao Ming
> On 2009-11-17 07:23, Lao Ming wrote:
>
>
>
>
>
> > On Nov 16, 10:11 pm, Ben Finney <bignose+hates-s...@benfinney.id.au>
> > wrote:
> >> Lao Ming <laoming...@gmail.com> writes:
> >>> If the page is modified, it still has the same URL. It is the actual
> >>> unique URL that I want to put a decimal value on and not the contents.
> >>> Thanks.
> >> In that case, simply generate a cryptographic hash of the URL and use
> >> the resulting number for the unique identifier. An SHA-1 hash returns a
> >> number, and should be adequate for what you've described so far.
>
> >> --
> >> \ “A free society is one where it is safe to be unpopular.” |
> >> `\ —Adlai Ewing Stevenson |
> >> _o__) |
> >> Ben Finney
>
> > Thanks. It's perfect! I don't know why I didn't think of that. I
> > guess I have always just used md5 with filenames not thinking it could
> > be used with any string. I don't have sha1sum but I could use:
>
>
> > Thanks again!
>
> you can also create a tinyurl for each url and make that to a number some way.http://tinyurl.com/
>
> /bb
How would one feed a URL to tinyURL or bit.ly from the shell and have
either site return its newly-formed URL?
Glenn Jackman
> On Nov 17, 4:03?am, bb <spamt...@norsborg.net> wrote:
> > you can also create a tinyurl for each url and make that to a number some way.http://tinyurl.com/
> >
> either site return its newly-formed URL?
There's a Perl module just for this:
http://search.cpan.org/~davecross/WWW-Shorten-2.05/lib/WWW/Shorten/TinyURL.pm
--
Glenn Jackman
Write a wise saying and your name will live forever. -- Anonymous
Lao Ming
wrote:
> 2009-11-16, 22:23(-08), Lao Ming:
> [...]> Thanks. It's perfect! I don't know why I didn't think of that. I
> > guess I have always just used md5 with filenames not thinking it could
> > be used with any string. I don't have sha1sum but I could use:
>
>
> [...]
>
> You may want to canonify the URL beforehand as
>
>
> Are all the same URL but have different MD5 sums.
>
> http://www.CNN.com:80/#some-fragment
>
> also points to the same page.
>
>
> and " " vs "+" vs "%20".
>
> perl (at least) has some modules for URI handling (and also for
> MD5 or SHA1 hashes)..
>
> Also note that in that command line above, you're including a
> trailing "\n" character in the string to hash.
>
> --
Thanks a bunch, Stéphane, for all the cautions. I think this bash
code would cover most of those. Since all of my URLs are filtered to
a degree by OSX (when e.g. it replaces all spaces with %20), I
probably won't have to concern myself with spaces and other characters
represented by %[0-F][0-F][0-F]. I could be wrong about that
though.
protocol=$( echo "$url" |awk -F':' '{print $1}' )
#
# extract and lowercase the dns_name
#
dns_name=$( echo "$url" |awk -F'/' '{print $3}' |tr '[A-Z]' '[a-
z]')
dns_name=${dns_name%:*} # remove any specified port
part1="${protocol}://$dns_name" # form part1 of url
part2="${url//$part1}" # form part2 by removing part1
from url
[ "$part2" = "/" ] && part2="" # remove any lone trailing slash
# URLs to inbound anchor should probably be preserved
#fragment=$( echo "$part2" |cut -c1-2 ) # extract first 2 chars
of part2
#[ "$fragment" = "/#" ] && part2="" # if fragment = fragment id, rn
part2
# reconstruct the URL to obtain an md5 digest
md5=$( printf "${part1}$part2" |md5 )
Wayne
I'm curious why you need to do this. The (normalized) URL is already
a unique identifier for static web pages. Do you need the number to
use as in index for a list or table, in a language that doesn't
permit strings as indexes?
If you use a smaller number than the max URL length the possibility
of collisions is remote put non-zero.
If you try to use dynamic pages, you end up with many different web
pages that have the same URL. Unless you also include a hash of
the contents at a URL, just the URL isn't a unique reference to
some particular content. Sadly a hash won't work either since
even static web pages sometimes contain processed output that changes
over time. I guess it depends on your definition of a unique "page".
Note if you also plan on supporting the new Unicode URLs, (ICANN
recently approved the use of Unicode for TLDs), you will need to
normalize the IRIs with punycode first.
(By the way, an IP address already is a unique number. Maybe you're
looking for a way to convert a dotted decimal encoding to a plain
decimal number?)
--
Wayne
Stephane CHAZELAS
[...]
> protocol=$( echo "$url" |awk -F':' '{print $1}' )
protocol=${url%%:*}
> #
> # extract and lowercase the dns_name
> #
> dns_name=$( echo "$url" |awk -F'/' '{print $3}' |tr '[A-Z]' '[a-
> z]')
> dns_name=${dns_name%:*} # remove any specified port
What about
http://user@host:port
http://user:password@host:port
> part1="${protocol}://$dns_name" # form part1 of url
> part2="${url//$part1}" # form part2 by removing part1
not fool proof either.
> from url
> [ "$part2" = "/" ] && part2="" # remove any lone trailing slash
That changes the URL http://foo/bar is often not the same as
http://foo/bar/
> # URLs to inbound anchor should probably be preserved
> #fragment=$( echo "$part2" |cut -c1-2 ) # extract first 2 chars
> of part2
> #[ "$fragment" = "/#" ] && part2="" # if fragment = fragment id, rn
> part2
>
> # reconstruct the URL to obtain an md5 digest
> md5=$( printf "${part1}$part2" |md5 )
printf %s "$part1$part2"
especially considering that those are likely to contain %
characters.
$ perl -MURI -le 'print URI->new("http://u\@host:80//a/b%2E%2F/./a+b?%20asd=http://www.google.com/+w#a/")->canonical'
http://u@host//a/b.%2F/./a+b?%20asd=http://www.google.com/+w#a/
--
Stï¿œphane
Barry Margolin
Wayne <nos...@all.4me.invalid> wrote:
> (By the way, an IP address already is a unique number. Maybe you're
> looking for a way to convert a dotted decimal encoding to a plain
> decimal number?)
The IP might be TOO unique. Many web sites are distributed among
multiple servers, and the IP changes when you look it up at different
times or from different locations. But for most purposes they should be
considered the same site.
--
Barry Margolin, bar...@alum.mit.edu
Arlington, MA
*** PLEASE post questions in newsgroups, not directly to me ***
*** PLEASE don't copy me on replies, I'll read them in the group ***
Lao Ming
> Lao Ming wrote:
> > Suppose I want to assign a unique number to any website anywhere --
> > would it be possible to obtain such a number such that any time I ran
> > my script, it would always determine a unique number for that very
> > page (as long as the web page exists)? If so, how could this be done?
>
> > I don't recall the math used but there used to be a way to get unique
> > decimal value for an IP address (it used to be calculated by MacTCP)
> > but it doesn't seem possible to obtain a value for any filepath. If
> > anyone has any ideas about this, I'd really appreciate it. Thanks.
>
> I'm curious why you need to do this. The (normalized) URL is already
> a unique identifier for static web pages. Do you need the number to
> use as in index for a list or table, in a language that doesn't
> permit strings as indexes?
I need a unique string to use as a filename for a snapshot of the web
page and the normalized URL wouldn't work unless I modified all the
punctuation to use a hex character code like %20. I would rather have
a name that more accurately reflects the content of the snapshot but I
can't think of anything else except perhaps extracting the hostname
and appending, say, part or all of the md5 hash.
> If you use a smaller number than the max URL length the possibility
> of collisions is remote put non-zero.
>
> If you try to use dynamic pages, you end up with many different web
> pages that have the same URL. Unless you also include a hash of
> the contents at a URL, just the URL isn't a unique reference to
> some particular content. Sadly a hash won't work either since
> even static web pages sometimes contain processed output that changes
> over time. I guess it depends on your definition of a unique "page".
I don't think I have to worry about that since the number of snapshots
is not likely to ever be that large of a number.
> Note if you also plan on supporting the new Unicode URLs, (ICANN
> recently approved the use of Unicode for TLDs), you will need to
> normalize the IRIs with punycode first.
That might be a problem if md5 cannot create a hash of a Unicode URL.
>
> (By the way, an IP address already is a unique number. Maybe you're
> looking for a way to convert a dotted decimal encoding to a plain
> decimal number?)
No. My reference to that was just because I didn't think of something
like md5 initially.
>
> --
> Wayne
Lao Ming
wrote:
> 2009-11-17, 21:28(-08), Lao Ming:
> [...]
>
> > protocol=$( echo "$url" |awk -F':' '{print $1}' )
>
> protocol=${url%%:*}
Cool!
> > #
> > # extract and lowercase the dns_name
> > #
> > dns_name=$( echo "$url" |awk -F'/' '{print $3}' |tr '[A-Z]' '[a-
> > z]')
> > dns_name=${dns_name%:*} # remove any specified port
>
> What about
>
> http://user@host:port
> http://user:password@host:port
Since the URLs I'm using will never have authentication to deal with,
that won't be a problem.
>
> > part1="${protocol}://$dns_name" # form part1 of url
> > part2="${url//$part1}" # form part2 by removing part1
>
> not fool proof either.
Why not?
> > from url
> > [ "$part2" = "/" ] && part2="" # remove any lone trailing slash
>
> That changes the URL
> http://foo/baris often not the same ashttp://foo/bar/
You may have read it incorrectly. part2 in the above case is a single
forward slash which was just to eliminate the duplication you proposed
for
and
>
> > # URLs to inbound anchor should probably be preserved
> > #fragment=$( echo "$part2" |cut -c1-2 ) # extract first 2 chars
> > of part2
> > #[ "$fragment" = "/#" ] && part2="" # if fragment = fragment id, rn
> > part2
>
> > # reconstruct the URL to obtain an md5 digest
> > md5=$( printf "${part1}$part2" |md5 )
>
> printf %s "$part1$part2"
>
> especially considering that those are likely to contain %
> characters.
Thanks.
>
> $ perl -MURI -le 'print URI->new("http://u\@host:80//a/b%2E%2F/./a+b?%20asd=http://www.google.com/+w#a/")->canonical'http://u@host//a/b.%2F/./a+b?%20asd=http://www.google.com/+w#a/
Thanks again for all the great ideas, Stéphane.
By the way, is there a reason that perl won't let me use a variable
for the URL?
For example, I tried:
URL="http://u\@host:80//a/b%2E%2F/./a+b?%20asd=http://www.google.com/
+w#a/"
and
perl -MURI -le 'print URI->new(\" $URL \")->canonical'
perl -MURI -le 'print URI->new(" $URL ")->canonical'
perl -MURI -le 'print URI->new("$URL")->canonical'
among many other similar attempts all failed.
>
> --
> Stéphane
Alan Curry
Lao Ming <laomi...@gmail.com> wrote:
>By the way, is there a reason that perl won't let me use a variable
>for the URL?
>
>For example, I tried:
>
>URL="http://u\@host:80//a/b%2E%2F/./a+b?%20asd=http://www.google.com/
>+w#a/"
>
>and
>
> perl -MURI -le 'print URI->new(\" $URL \")->canonical'
> perl -MURI -le 'print URI->new(" $URL ")->canonical'
> perl -MURI -le 'print URI->new("$URL")->canonical'
>
>among many other similar attempts all failed.
The '' quoting around the perl script prevents shell variable expansion.
There are at least 3 easy ways around it:
1. Use an argument
perl -MURI -le 'print URI->new($ARGV[0])->canonical' "$URL"
2. Use the environment
U="$URL" perl -MURI -le 'print URI->new($ENV{U})->canonical'
3. Use a pipe
echo "$URL" | perl -MURI -nle 'print URI->new($_)->canonical'
--
Alan Curry
Kaz Kylheku
0. Too obvious:
Kaz Kylheku
0. a) Indeed, too:
Alan Curry
URL='http://en.wikipedia.org/wiki/$'
URL='http://en.wikipedia.org/wiki/$'
echo -n '1. '
perl -MURI -le 'print URI->new($ARGV[0])->canonical' "$URL"
echo -n '2. '
U="$URL" perl -MURI -le 'print URI->new($ENV{U})->canonical'
echo -n '3. '
echo "$URL" | perl -MURI -nle 'print URI->new($_)->canonical'
echo -n '0. '
perl -MURI -le 'print URI->new("'$URL'")->canonical'
1. http://en.wikipedia.org/wiki/$
2. http://en.wikipedia.org/wiki/$
3. http://en.wikipedia.org/wiki/$
0. Final $ should be \$ or $name at -e line 1, within string
syntax error at -e line 1, near "("http://en.wikipedia.org/wiki/$""
Execution of -e aborted due to compilation errors.
(Moderately surprised that it didn't canonicalize to %24)
--
Alan Curry
bb
Try it by hand first and make a tiny url from a long one.
Then you can use curl to do the job for you.
Maybe they have some help or examples at the tinyurl site.
/bb
bb
What about storing your snapshot as yhfztcx.png ?
And it represent a snapshot of:
http://tinyurl.com/yhfztcx
/bb