Wrong SHA1 is calculated

[Martin Puppe]

Hello,

 

I am debugging a problem with SHA1 checksums. I have found the following handy one-liner on Stack Overflow [^1], which serves well as a minimal example:

 

```

perl -MDigest::SHA1=sha1_hex -le "print sha1_hex <>" secure.txt

```

 

The problem is, that the result is simply not correct. Doing the same on a Unix system gives the correct result. And I also get a correct result when I generate the SHA1 sum with the 7-zip file manager.

 

Any ideas what the problem might be?

 

[^1]: http://stackoverflow.com/a/2812212

 

Best regards,

Martin Puppe

JAM Software

---

--------------------------------------------------------
JAM Software GmbH
Managing Director: Joachim Marder
Am Wissenschaftspark 26 * 54296 Trier * Germany
Phone: +49 (0)651-145 653 -0 * Fax: +49 (0)651-145 653 -29
Commercial register number HRB 4920 (AG Wittlich) http://www.jam-software.com

---

----------------------------------------------------
JAM Software GmbH
Geschäftsführer: Joachim Marder
Am Wissenschaftspark 26 * 54296 Trier * Germany
Tel: 0651-145 653 -0 * Fax: 0651-145 653 -29
Handelsregister Nr. HRB 4920 (AG Wittlich) http://www.jam-software.de

[sisy...@optusnet.com.au]

From: Martin Puppe
Sent: Monday, May 15, 2017 8:39 PM
To: win32-...@perl.org
Subject: Wrong SHA1 is calculated

> Hello,
>
> I am debugging a problem with SHA1 checksums. I have found the following
> handy one-liner on Stack Overflow [^1], which serves well as a minimal
> example:
>
> perl -MDigest::SHA1=sha1_hex -le "print sha1_hex <>" secure.txt
>

> The problem is, that the result is simply not correct.Martin Puppe

The same web page presents a program which should provide the same "not
correct" result.
I've inserted "binmode $fh;" into it - which then allows it to return the
"correct" value:

#############################
use warnings;
use strict;

use Digest::SHA1;

die "Usage: $0 file ..\n" unless @ARGV;

foreach my $file (@ARGV) {
my $fh;
unless (open $fh, $file) {
warn "$0: open $file: $!";
next;
}

binmode $fh; # inserted by sisyphus
my $sha1 = Digest::SHA1->new;
$sha1->addfile($fh);
print $sha1->hexdigest, " $file\n";

close $fh;
}
###############################

If 'secure.txt' is a plain text file with Unix line endings, then that
binmode() should not be necessary - but if the text file has Windows line
endings then binmode() will prevent their translation to Unix endings (and
the program will return the result that you expect).

Unfortunately, I don't know how to get that binmode() into the one-liner's
angle brackets :-(

Maybe someone else here can chime in.

Cheers,
Rob

[Christian Millour]

Le 15/05/2017 à 16:30, sisy...@optusnet.com.au a écrit :
> Unfortunately, I don't know how to get that binmode() into the
> one-liner's angle brackets :-(

you might consider playing with the PERLIO environment variable :

$ perl -E "print qq{hello\nworld\n}" > secure.txt
$ od -c secure.txt
0000000 h e l l o \r \n w o r l d \r \n
0000016
$ perl -MDigest::SHA1=sha1_hex -le "print sha1_hex <>" secure.txt
58853e8a5e8272b1012f9a52a80758b27bd0d3cb
$ PERLIO=unix perl -MDigest::SHA1=sha1_hex -le "print sha1_hex <>"
secure.txt
19576d392b021ac25efdca6f1886b5ce5b1090c4
$

--Christian

[sisy...@optusnet.com.au]

-----Original Message-----
From: Christian Millour
Sent: Tuesday, May 16, 2017 9:14 AM
To: win32-...@perl.org
Subject: Re: Wrong SHA1 is calculated

> Le 15/05/2017 à 16:30, sisy...@optusnet.com.au a écrit :
>> Unfortunately, I don't know how to get that binmode() into the
>> one-liner's angle brackets :-(

> $ PERLIO=unix perl -MDigest::SHA1=sha1_hex -le "print sha1_hex <>"
> secure.txt
> 19576d392b021ac25efdca6f1886b5ce5b1090c4
> $

Yes, I think that should give the OP the result he was seeking.
Nice.

Cheers,
Rob

[sisy...@optusnet.com.au]

>> Le 15/05/2017 à 16:30, sisy...@optusnet.com.au a écrit :
>>> Unfortunately, I don't know how to get that binmode() into the
>>> one-liner's angle brackets :-(
>>
>> $ PERLIO=unix perl -MDigest::SHA1=sha1_hex -le "print sha1_hex <>"
>> secure.txt
>> 19576d392b021ac25efdca6f1886b5ce5b1090c4
>

> Yes, I think that should give the OP the result he was seeking.

Heh ... I didn't realize (until after I had posted) that this was a bash
shell solution.
In the cmd shell, of course, the same command simply outputs:

'PERLIO' is not recognized as an internal or external command,
operable program or batch file.

Best approximation I could come up with in the cmd shell was:

set PERLIO=unix && perl -MDigest::SHA1=sha1_hex -le "print sha1_hex <>"
secure.txt && set PERLIO=
19576d392b021ac25efdca6f1886b5ce5b1090c4

which is not quite so nice as in bash ;-)

Can my cmd shell implementation be improved upon ?

Cheers,
Rob

[Christian Millour]

Le 16/05/2017 à 13:11, sisy...@optusnet.com.au a écrit :
>>> Le 15/05/2017 à 16:30, sisy...@optusnet.com.au a écrit :
>>>> Unfortunately, I don't know how to get that binmode() into the
>>>> one-liner's angle brackets :-(
>>>
>>> $ PERLIO=unix perl -MDigest::SHA1=sha1_hex -le "print sha1_hex <>"
>>> secure.txt
>>> 19576d392b021ac25efdca6f1886b5ce5b1090c4
>>
>> Yes, I think that should give the OP the result he was seeking.
>
> Heh ... I didn't realize (until after I had posted) that this was a bash
> shell solution.

Yes, the intent was only to draw attention to the PERLIO environment
variable as a possible alternative to binmode, not to offer a complete
solution.

OTOH this was actually run on windows using Strawberry perl from a
cygwin bash shell. That has been my standard setup on windows for more
years than I care to count. Given that you can setup a decent portable
environment including cygwin, ntemacs (with cygwin bash as shell and
tramp via putty for remote anything), strawberry perl(s), and your own
minicpan/darkpan, all on a USB key, that works on any windows from 2000
to win10, I have seen no need to come close to CMD for quite some time :-)

BTW, the :crlf layer is hopelessly broken on windows (can't seek
reliably) so be sure to binmode any file you need to navigate with
anything other than the angle brackets.

The PERLIO variable has also been a lifesaver on some instance where
CPAN heroes, saintly as they are, could not find it in their heart to
test their modules updates on a windows box. If you ever tried to e.g.
install dzil in its early days you will know :-) In such cases it
actually makes sense to setup a whole environment with a nonstandard
setting of PERLIO. YMMV.

Hmmm. I should probably note that because of compatibility issues
between cygwin and ntemacs (if you want to run the cygwin bash as your
emacs shell), I run a very old version of cygwin, probably older than
2009 (its perl at the time was 5.10 I believe). Its bash and make are
sufficient for my need, and I have no trouble using various instances of
strawberry perl with it. No doubt others will have different standards
so take the above with a pinch of salt.

Regards,

--Christian

[Martin Puppe]

Thank you very much! That was very helpful in debugging a problem with SpamAssassin's sa-update.

Best regards,
Martin

-----Ursprüngliche Nachricht-----
Von: sisy...@optusnet.com.au [mailto:sisy...@optusnet.com.au]
Gesendet: Montag, 15. Mai 2017 16:31
An: Martin Puppe <pu...@jam-software.com>; win32-...@perl.org
Betreff: Re: Wrong SHA1 is calculated